[HADOOP-17074] Optimise s3a Listing to be fully asynchronous.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Assignee: Mukund Thakur
• Reporter: Mukund Thakur
• Created: 17/Jun/20 12:31
• Updated: 30/Nov/20 10:37
• Resolved: 25/Aug/20 10:32

Description

Listing api returns a Remote iterator. Currently it is blocking in the constructor Listing class. Also it is done in batches, so if a batch is exhausted , the next batch will be fetched from s3 

thus causing small pauses on the client side every time a batch is fetched. We can add the following optimisations to fix these:

• Make the constructor list call to s3 asynchronous.
• Fetch the next batch asynchronously while the current batch is getting processed.

Issue Links

• is related to: HADOOP-17335 s3a listing operation will fail in async prefetch if fs closed
• links to: PR 2207