[HADOOP-13062] S3A Introspect to invoke incompatible AWS TransferManagerConfiguration methods

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.7.2
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 26/Apr/16 10:19
• Updated: 06/May/16 18:12
• Resolved: 06/May/16 18:12

Description

The AWS SDK changed the signature of the TransferManagerConfiguration.setMultipartUploadThreshold, moving one param from an int to a long. This is fixed at compile time, so S3a built against the old library doesn't link to to the new one, and vice versa —something leading to problems downstream.

It may be possible to use reflection to make this binding, at least on the 2.7 branch, so that dropping in to a later SDK doesn't break things

Issue Links

• is related to: HADOOP-13060 While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V