Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16080

hadoop-aws does not work with hadoop-client-api

    XMLWordPrintableJSON

    Details

      Description

      I attempted to use Accumulo and S3a with the following jars on the classpath.

      • hadoop-client-api-3.1.1.jar
      • hadoop-client-runtime-3.1.1.jar
      • hadoop-aws-3.1.1.jar

      This failed with the following exception.

      Exception in thread "init" java.lang.NoSuchMethodError: org.apache.hadoop.util.SemaphoredDelegatingExecutor.<init>(Lcom/google/common/util/concurrent/ListeningExecutorService;IZ)V
          at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:769)
          at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
          at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
          at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1108)
          at org.apache.hadoop.fs.FileSystem.createNewFile(FileSystem.java:1413)
          at org.apache.accumulo.server.fs.VolumeManagerImpl.createNewFile(VolumeManagerImpl.java:184)
          at org.apache.accumulo.server.init.Initialize.initDirs(Initialize.java:479)
          at org.apache.accumulo.server.init.Initialize.initFileSystem(Initialize.java:487)
          at org.apache.accumulo.server.init.Initialize.initialize(Initialize.java:370)
          at org.apache.accumulo.server.init.Initialize.doInit(Initialize.java:348)
          at org.apache.accumulo.server.init.Initialize.execute(Initialize.java:967)
          at org.apache.accumulo.start.Main.lambda$execKeyword$0(Main.java:129)
          at java.lang.Thread.run(Thread.java:748)
      

      The problem is that S3AFileSystem.create() looks for SemaphoredDelegatingExecutor(com.google.common.util.concurrent.ListeningExecutorService) which does not exist in hadoop-client-api-3.1.1.jar. What does exist is SemaphoredDelegatingExecutor(org.apache.hadoop.shaded.com.google.common.util.concurrent.ListeningExecutorService).

      To work around this issue I created a version of hadoop-aws-3.1.1.jar that relocated references to Guava.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                csun Chao Sun
                Reporter:
                kturner Keith Turner
              • Votes:
                1 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h 10m
                  8h 10m