Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13062

S3A Introspect to invoke incompatible AWS TransferManagerConfiguration methods

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 2.7.2
    • None
    • fs/s3
    • None

    Description

      The AWS SDK changed the signature of the TransferManagerConfiguration.setMultipartUploadThreshold, moving one param from an int to a long. This is fixed at compile time, so S3a built against the old library doesn't link to to the new one, and vice versa —something leading to problems downstream.

      It may be possible to use reflection to make this binding, at least on the 2.7 branch, so that dropping in to a later SDK doesn't break things

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-13060 While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: