Currently, the JHS scan directory if the modification of directory changed:
This logic relies on an assumption that, the directory's modification time will be updated if a file got placed under the directory.
However, the semantic of directory's modification time is not consistent in different FS implementations. For example,
MAPREDUCE-6680 fixed some issues of truncated modification time. And HADOOP-12837 mentioned on S3, the directory's modification time is always 0.
I think we need to revisit behavior of this logic to make it to more robustly work on different file systems.