[HADOOP-17275] Implement Incremental globStatus() -> RemoteIterator class

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: fs
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 18/Sep/20 12:44
• Updated: 27/Jul/22 16:11

Description

globStatus/globber is inefficient, not incremental and doesn't provide a way to query any IOStatistics.

Proposed:

• new IncrementalGlobber class which takes an interface to the file source (+impls for FS and FC) which offer the incremental ops
• use incremental listStatusIncremental calls
• or, if no wildcards, full deep listFiles(recursive)
• collect IOStatistics from source ops, aggregate and serve

Potentially a lot of work...need to look carefully here