Java stream - too many open files -
i have program goes few directories time time , and kind of processing files in directories.
the problem time time (each 2 or 3 days) program reaching os open files limit.
it spring-boot application running in rhel 7.
the method files this:
public file[] getfiles(string dir, int numberoffiles) throws exception { final path basedir = paths.get(dir); list<file> filesfrompath = new arraylist<file>(); file[] files = null; final bipredicate<path, basicfileattributes> predicate = (path, attrs) -> attrs.isregularfile() && string.valueof(path).endswith(".xml"); list<path> result; try (stream<path> filestream = files.find(basedir, 1, predicate).limit(numberoffiles).onclose(() -> log.debug("closing file stream."))){ result = filestream.collect(collectors.tolist()); result.foreach(path -> { path.tostring(); file file = path.tofile(); log.info("adding {} process.", file.getname()); filesfrompath.add(file); }); if (filesfrompath != null && !filesfrompath.isempty()) { files = filesfrompath.toarray(new file[filesfrompath.size()]); } } catch (exception e) { log.error("error during file opening/closing", e); } if (files != null) { return files; } return new file[0]; }
i using lsof command see how many opened files have, , list of directories growing.
i added log in onclise method , being called times open stream.
shouldn't try resources, close stream?
[edit]
there peace of code move processed files folder. code not use stream, , couldn't found out wrong it, besides fact ugly.
public void move(file file, string archivepath) throws ioexception { file backupfile = new file(archivepath); if (!backupfile.exists()) { backupfile.mkdirs(); } path source = file.topath(); if (file.exists()) { path target = paths.get(archivepath + file.separator + file.getname()); files.move(source, target, standardcopyoption.replace_existing); log.info("file {} moved {}", file, archivepath); } else { log.info("unable move file: {} because moved {}", file, archivepath); } }
[edit 2]
and files being processed this:
private void processfile(file[] files) throws exception { documentbuilderfactory dbf = documentbuilderfactory.newinstance(); documentbuilder db = dbf.newdocumentbuilder(); xpath xpathparser = xpathfactory.newinstance().newxpath(); (int = 0; < files.length; i++) { file file = files[i]; document doc = db.parse(file); // stuff fileutils.move(file, processedpath); } }
thanks.
Comments
Post a Comment