[SPARK-49508] Optimized hadoop-aws dependency, aws-java-sdk-bundle jar is too large

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0, 3.5.2
• Component/s: Build
• Assignee: Unassigned
• Reporter: melin
• Created: 04/Sep/24 04:42
• Updated: 18/Oct/24 16:28

Description

aws-java-sdk-bundle jar is too large，The size of the spark image will double。hadoop aws only requires the use of aws-java-sdk-s3 and aws-java-sdk-dynamodb

 

// code placeholder
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-aws</artifactId>
    <version>${hadoop.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-bundle</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>${awssdk.v1.version}</version>
</dependency>
<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-dynamodb</artifactId>
    <version>${awssdk.v1.version}</version>
</dependency>