Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
3.3.4
-
None
-
None
Description
ViewFileSystem allows a federated view of a file system, so that for example under the path foo/ I might have foo/bar1 mapped to some other file system, foo/bar2 mapped to some different file system, etc. using the ViewFS mount table.
Consider a situation where I have 1,000 subdirectories foo/bar000 to foo/bar999 mapped to 1,000 different cloud providers (e.g. AWS S3 buckets or whatever). Let's say that for whatever reason the mapping for foo/bar123 was incorrect (maybe there was a corrupted mount table or a race condition in creating the destination cloud storage), so that when we we try to get the status of foo/bar123 it returns an HTTP 404, throwing an exception.
But let's say that we were instead listing the status of foo/ itself, in order to return all 1,000 children. Look what would happen in the ViewFileSystem.listStatus(Path f) code when we call ViewFileSystem.listStatus(new Path("…/foo")). We expect it to return 999 child paths instead of 1,000 child (because one of the mounted paths is misconfigured and returns 404)):
for (Entry<String, INode<FileSystem>> iEntry :
theInternalDir.getChildren().entrySet()) {
…
try {
FileStatus status =
((ChRootedFileSystem)link.getTargetFileSystem())
.getMyFs().getFileStatus(new Path(linkedPath));
linkStatuses.add(
new FileStatus(status.getLen(), status.isDirectory(),
status.getReplication(), status.getBlockSize(),
status.getModificationTime(), status.getAccessTime(),
status.getPermission(), status.getOwner(),
status.getGroup(), null, path));
} catch (FileNotFoundException ex) {
LOG.warn("Cannot get one of the children's(" + path
+ ") target path(" + link.getTargetFileSystem().getUri()
+ ") file status.", ex);
throw ex;
}
For each particular child that is mapped in the map table, a ((ChRootedFileSystem)link.getTargetFileSystem()).getMyFs().getFileStatus(new Path(linkedPath)) is performed on the underlying federated file system and the resulting `FileSystatus` is added to the list. But in the case of foo/bar123, it throws an exception. The code above appropriately catches the exception and warns, "Cannot get one of the children's … file status" That part is perfectly fine. But then the code rethrows the exception, which is incorrect.
Rethrowing the exception with throw ex breaks the directory listing; it will result in an exception for the entire directory listing of foo/, not just the child. If the child mapping for foo/bar123 has somehow disappeared (maybe it's just a race condition, and that the mapping table was stale when the directory listing started so that the mapping was never current) and foo/bar123 returns a 404, suddenly the entire directory listing, instead of returning 999 entries as expected doesn't return any entries because the file status listing of foo/ itself returns 404!
This bug essentially causes an entire subtree to disappear merely because of a problem accessing one of the children. In a distributed environment (which is what ViewFs was intended for), with thousands of mappings to various HTTP-based cloud storage accounts, it's not unexpected that one of them might be temporarily unavailable. But this bug would cause the parent directory to seem unavailable, essentially making it appear that e.g. /users simply did not exist simply because /users/fulano happened to be missing.
And if we happen to have /missing-mount mounted under the root and it was temporarily unavailable, and we did a listStatus() on the root directory / itself? Yes, it would appear as if the root directory itself was missing, i.e. the entire federated file system.
I have seen this bug in practice. In fact I had thought I had already filed a ticket for this, but maybe it was at some organization's internal bug tracking system instead of on the public Apache Hadoop bug tracking system.
You can verify this bug simply by adding a unit/integration test that mocks foo/bar1, foo/bar2, and foo/bar3 as ChRootedFileSystem in a ViewFileSystem via ViewFileSystem.getMyFs(). Perform a ViewFileStatus.listStatus() on foo/ and see that it returns 3 children. Then have getMyFs().getFileStatus() return a 404 error only for foo/bar2. Do a ViewFileStatus.listStatus() on foo/ again, and instead of returning 2 children, it will claim that foo/ itself does not exist.
Fixing this bug is very simple: remove the throw ex altogether on line 1449.