[HADOOP-15387] Produce a shaded hadoop-cloud-storage JAR for applications to use

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: fs/adl, fs/azure, fs/oss, fs/s3, fs/swift
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 16/Apr/18 09:49
• Updated: 04/Jan/24 12:27

Description

Produce a maven-shaded hadoop-cloudstorage JAR for dowstream use so that

• Hadoop dependency choices don't control their decisions
• Little/No risk of their JAR changes breaking Hadoop bits they depend on

This JAR would pull in the shaded hadoop-client JAR, and the aws-sdk-bundle JAR, neither of which would be unshaded (so yes, upgrading aws-sdks would be a bit risky, but double shading a pre-shaded 30MB JAR is excessive on multiple levels.

Metrics of success: Spark, Tez, Flink etc can pick up and use, and all are happy

Issue Links

• is related to: HADOOP-15924 Hadoop aws does not use shaded jars
• is related to: HADOOP-16080 hadoop-aws does not work with hadoop-client-api