> I have concerns about recursive listing. This could be abused by the applications, creating a lot of requests into HDFS.
We could put this new API in FileContext.utils. FileSystem will be deprecated soon so no need to worry about it.
> Any deletion of files/directories while reursing through directories results in RuntimeException and application has a partial result.
> Should we ignore if a directory was in stack and was not found later when iterating through it?
GetFiles is used by MapReduce job client. In the scenario that you described, it is good to throw an exception so fail the job earlier. Otherwise, the job won't fail until map tasks are launched.
I will incorporate the rest of the comments.