[HADOOP-17475] ABFS : add high performance listStatusIterator

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.1
• Component/s: fs/azure
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Assignee: Bilahari T H
• Reporter: Bilahari T H
• Created: 18/Jan/21 04:34
• Updated: 13/Jul/21 14:30
• Resolved: 04/Feb/21 13:50

Description

The ABFS connector now implements listStatusIterator() with
asynchronous prefetching of the next page(s) of results.
For listing large directories this can provide tangible speedups.

If for any reason this needs to be disabled, set
fs.azure.enable.abfslistiterator to false.

Issue Links

• is depended upon by: HADOOP-17512 Remove the enable/disable flag for ABFSRemoteListIterator
• is related to: HADOOP-17797 transient failure of ITestAbfsListStatusRemoteIterator.testWithAbfsIteratorDisabledWithoutHasNext
• links to: GitHub Pull Request #2548
• links to: IOStatistics collection for listStatusIterator()