[HADOOP-13060] While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Works for Me
• Affects Version/s: 2.7.1
• Fix Version/s: 2.7.3
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Jenny Dong
• Created: 25/Apr/16 21:11
• Updated: 28/Apr/17 09:52
• Resolved: 28/Apr/17 09:52

Description

While trying to do a distcp from our native HDFS cluster to S3, we get the following error/stacktrace :

We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors seen in HADOOP-12420 + other authentication errors).

Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)

I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem created a ThreadPoolExecutor (which implements AbstractExecutorService which implements ExecutorService). I also checked on the classpath to make sure the version of the jars being picked up is what I expected.

Help would be much appreciated.

Issue Links

• relates to: HADOOP-13062 S3A Introspect to invoke incompatible AWS TransferManagerConfiguration methods