Listing api returns a Remote iterator. Currently it is blocking in the constructor Listing class. Also it is done in batches, so if a batch is exhausted , the next batch will be fetched from s3
thus causing small pauses on the client side every time a batch is fetched. We can add the following optimisations to fix these:
- Make the constructor list call to s3 asynchronous.
- Fetch the next batch asynchronously while the current batch is getting processed.
- is related to
HADOOP-17335 s3a listing operation will fail in async prefetch if fs closed
- links to