[HADOOP-17337] S3A NetworkBinding has a runtime class dependency on a third-party shaded class

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1
• Component/s: fs/s3
• Labels: pull-request-available
• Release Note: the s3a filesystem will link against the unshaded AWS s3 SDK. Making an application's dependencies consistent with that SDK is left as exercise. Note: native openssl is not supported as a socket factory in unshaded deployments.
• Assignee: Steve Loughran
• Reporter: Chris Wensel
• Created: 30/Oct/20 20:44
• Updated: 01/Jun/21 16:50
• Resolved: 03/Feb/21 14:35

Description

The hadoop-aws library has a dependency on 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS SDK libraries and shaded dependencies.

 

This dependency is 181MB.

 

Some applications using the S3AFilesystem may be sensitive to having a large footprint. For example, building an application using Parquet and bundled with Docker.

 

Typically, in prior Hadoop versions, the bundle was replaced by the specific AWS SDK dependencies, dropping the overall footprint.

 

In 3.3 (and maybe prior versions) this strategy does not work because of the following exception: 

java.lang.NoClassDefFoundError: com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory
{{ at org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
{{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
{{ at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
{{ at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
{{ at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
{{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
{{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
{{ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
{{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
{{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}

 

Issue Links

• is broken by: HADOOP-16371 Option to disable GCM for SSL connections when running on Java 8
• relates to: HADOOP-14040 Use shaded aws-sdk uber-JAR 1.11.86
• relates to: IMPALA-10073 Create shaded dependency for S3A and aws-java-sdk-bundle
• links to: GitHub Pull Request #2599