[HADOOP-17197] Decrease size of s3a dependencies

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Sahil Takiar
• Created: 10/Aug/20 17:22
• Updated: 11/Aug/20 09:44
• Resolved: 10/Aug/20 18:30

Description

S3A currently has a dependency on the aws-java-sdk-bundle, which includes the SDKs for all AWS services. The jar file for the current version is about 120 MB, but continues to grow (the latest is about 170 MB). Organic growth is expected as more and more AWS services are created.

The aws-java-sdk-bundle jar file is shaded as well, so it includes all transitive dependencies.

It would be nice if S3A could depend on smaller jar files in order to decrease the size of jar files pulled in transitively by clients. Decreasing the size of dependencies is particularly important for Docker files, where image pull times can be affected by image size.

One solution here would be for S3A to publish its own shaded jar which includes the SDKs for all needed AWS Services (e.g. S3, DynamoDB, etc.) along with the transitive dependencies for the individual SDKs.

Issue Links

• relates to: IMPALA-10028 Additional optimizations of Impala docker container sizes