Thanks Andrew for working on this, and Brahma for bringing up HDFS-9366. Patch itself looks good.
The feature the 2 jiras want to solve looks very alike - allowing some pagination when listing. Implementation is a little different:
HDFS-10784 implements RemoteIterator, so adds a new interface listStatusIterator.
- HDFS-9366 overloads current listStatus, with customized offset and size parameters.
HDFS-10784 is cleaner and more flexible, hence easier to use when user wants to iterate the whole listing. HDFS-9366 could have less end-to-end trips when listing with a starting offset not at the beginning.
Looking at DistributedFileSystem, an iterator would be more consistent with hdfs context. I don't have a strong opinion, maybe we should ask our user Romain Rigaux since both jira seems to aim at Hue.
I think we should combine this and HDFS-9366 after agreement, and add documentation. Would be great if httpfs is supported too.