Details
Description
I attempted to use Accumulo and S3a with the following jars on the classpath.
- hadoop-client-api-3.1.1.jar
- hadoop-client-runtime-3.1.1.jar
- hadoop-aws-3.1.1.jar
This failed with the following exception.
Exception in thread "init" java.lang.NoSuchMethodError: org.apache.hadoop.util.SemaphoredDelegatingExecutor.<init>(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108) at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413) at org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184) at org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479) at org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487) at org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370) at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348) at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967) at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129) at java.lang.Thread.run(Thread.java:748)
The problem is that S3AFileSystem.create() looks for SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService) which does not exist in hadoop-client-api-3.1.1.jar. What does exist is SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService).
To work around this issue I created a version of hadoop-aws-3.1.1.jar that relocated references to Guava.
Attachments
Issue Links
- causes
-
SPARK-33618 hadoop-aws doesn't work
-
- Resolved
-
- relates to
-
HADOOP-15924 Hadoop aws does not use shaded jars
-
- Resolved
-
-
HADOOP-17573 Fix compilation error of OBSFileSystem in trunk
-
- Resolved
-
-
HADOOP-15387 Produce a shaded hadoop-cloud-storage JAR for applications to use
-
- Open
-
- links to