[HADOOP-16080] hadoop-aws does not work with hadoop-client-api

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.1.1, 3.4.0
• Fix Version/s: 3.2.2, 3.3.1, 3.4.0
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.1.1, 3.2.2, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Chao Sun
• Reporter: Keith Turner
• Created: 28/Jan/19 22:05
• Updated: 10/Mar/21 10:18
• Resolved: 04/Dec/20 00:14

Description

I attempted to use Accumulo and S3a with the following jars on the classpath.

• hadoop-client-api-3.1.1.jar
• hadoop-client-runtime-3.1.1.jar
• hadoop-aws-3.1.1.jar

This failed with the following exception.

Exception in thread "init" java.lang.NoSuchMethodError: org.apache.hadoop.util.SemaphoredDelegatingExecutor.<init>(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
    at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
    at org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
    at org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
    at org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
    at org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
    at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
    at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
    at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
    at java.lang.Thread.run(Thread.java:748)

The problem is that S3AFileSystem.create() looks for SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService) which does not exist in hadoop-client-api-3.1.1.jar. What does exist is SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService).

To work around this issue I created a version of hadoop-aws-3.1.1.jar that relocated references to Guava.

Issue Links

• causes: SPARK-33618 hadoop-aws doesn't work
• relates to: HADOOP-15924 Hadoop aws does not use shaded jars
• relates to: HADOOP-17573 Fix compilation error of OBSFileSystem in trunk
• relates to: HADOOP-15387 Produce a shaded hadoop-cloud-storage JAR for applications to use
• links to: GitHub Pull Request #2510
• links to: GitHub Pull Request #2522
• links to: GitHub Pull Request #2575