[HADOOP-10987] Provide an iterator-based listing API for FileSystem

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.7.0
• Target Version/s: 2.7.0
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 20/Aug/14 18:01
• Updated: 10/Jun/16 15:43
• Resolved: 03/Nov/14 14:25

Description

Iterator based listing methods already exist in FileContext for both simple listing and listing with locations. However, FileSystem lacks the former. From what I understand, it wasn't added to FileSystem because it was believed to be phased out soon. Since FileSystem is very well alive today and new features are getting added frequently, I propose adding an iterator based listStatus method. As for the name of the new method, we can use the same name used in FileContext : listStatusIterator().

It will be particularly useful when listing giant directories. Without this, the client has to build up a huge data structure and hold it in memory. We've seen client JVMs running out of memory because of this.

Once this change is made, we can modify FsShell, etc. in followup jiras.

Issue Links

• is related to: HADOOP-13256 define FileSystem.listStatusIterator, implement contract tests
• relates to: HADOOP-11255 listLocatedStatus does not support cross-filesystem symlinks