Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-17337

S3A NetworkBinding has a runtime class dependency on a third-party shaded class

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.1
    • Component/s: fs/s3
    • Release Note:
      the s3a filesystem will link against the unshaded AWS s3 SDK. Making an application's dependencies consistent with that SDK is left as exercise. Note: native openssl is not supported as a socket factory in unshaded deployments.

      Description

      The hadoop-aws library has a dependency on 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS SDK libraries and shaded dependencies.

       

      This dependency is 181MB.

       

      Some applications using the S3AFilesystem may be sensitive to having a large footprint. For example, building an application using Parquet and bundled with Docker.

       

      Typically, in prior Hadoop versions, the bundle was replaced by the specific AWS SDK dependencies, dropping the overall footprint.

       

      In 3.3 (and maybe prior versions) this strategy does not work because of the following exception: 

      java.lang.NoClassDefFoundError: com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory
      {{ at org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}}
      {{ at org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}}
      {{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}}
      {{ at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}}
      {{ at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}}
      {{ at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}}
      {{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}}
      {{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}}
      {{ at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}}
      {{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}}
      {{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}}

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                cwensel Chris Wensel
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h
                  2h