[IO-755] Using FileUtils.listFiles() with background changes fails on Linux

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0, 2.10.0, 2.11.0
• Fix Version/s: 2.12.0
• Component/s: Utilities
• Labels: listFiles
• Assignee: Gary D. Gregory
• Reporter: Alexis Suter
• Created: 28/Oct/21 12:30
• Updated: 10/Oct/22 15:46
• Resolved: 20/Jun/22 13:12

Description

When calling FileUtils.listFiles and a file is being removed from this or a subdirectory during the execution the following Error occurs on linux.

Stacktrace

Unable to find source-code formatter for language: trace. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml

Caused by: java.nio.file.NoSuchFileException: /**
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
	at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
	at java.base/java.nio.file.Files.walkFileTree(Files.java:2761)
	at org.apache.commons.io.FileUtils.listAccumulate(FileUtils.java:2076)
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2132)

Upgrade of apache commons-io

The version of the dependency was upgraded from 2.8 to 2.11. In 2.9 the library started using java.nio instead of java.io which might behave differently on unix, when a file is not existent during processing of directories.

Causing functionality in our implementation

In some part we have multithreaded access to the filesystem. A folder is accessed from multiple threads which access the folder with org.apache.commons.io.FileUtils#listFiles(java.io.File, org.apache.commons.io.filefilter.IOFileFilter, org.apache.commons.io.filefilter.IOFileFilter). These threads also move, delete and rename files from this directory. While this seems to be working on a windows machine, it is not the case on a Linux machine. In 2.8 apparently there was no issue with this.

In short: While a directory is being listed with FileUtils.listFiles(), a file is being deleted. When trying to read its file-attributes it stops because of a NoSuchFileException.

Other commons-io functionalities

We tried using PathUtils.walk(directory filters, 5, false).map(Path::toFile).collect(Collectors.toList()). This method was introduced with the same change.
This produced a similar stacktrace:

Unable to find source-code formatter for language: trace. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml

Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /*
	at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
	at java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	... 47 more
Caused by: java.nio.file.NoSuchFileException: /*
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
	at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
	at java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:83)
	... 59 more

git commit

https://github.com/apache/commons-io/commit/4a514d3306b55b3667d1449ebd4cbe5f19dd7af0

Other Questions

• What was the reason to wrap all IOException into UncheckedIOException in those methods?