[HADOOP-16132] Support multipart download in S3AFileSystem

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Justin Uang
• Created: 21/Feb/19 16:09
• Updated: 04/Oct/22 16:23

Description

I noticed that I get 150MB/s when I use the AWS CLI

aws s3 cp s3://<bucket>/<key> - > /dev/null

vs 50MB/s when I use the S3AFileSystem

hadoop fs -cat s3://<bucket>/<key> > /dev/null

Looking into the AWS CLI code, it looks like the download logic is quite clever. It downloads the next couple parts in parallel using range requests, and then buffers them in memory in order to reorder them and expose a single contiguous stream. I translated the logic to Java and modified the S3AFileSystem to do similar things, and am able to achieve 150MB/s download speeds as well. It is mostly done but I have some things to clean up first. The PR is here: https://github.com/palantir/hadoop/pull/47/files

It would be great to get some other eyes on it to see what we need to do to get it merged.

Issue Links

• links to: GitHub Pull Request #645