Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-4611

Provide a migration path to commons-configuration2 to better support Hadoop 3

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.7.2, 1.8.1
    • Fix Version/s: 1.9.0, 2.0.0
    • Component/s: None
    • Environment:

      CentOS 7
      Accumulo 1.7.x, 1.8.x
      Hadoop 3.0.0-alpha 2
      Zookeeper 3.4.9

      Description

      I was investigating running either Accumulo 1.7.x or 1.8.x on Hadoop 3.0.0-alpha 2 and ran into a couple of issues. Since Accumulo assumes some dependencies will be provided by Hadoop (per ACCUMULO-1244), if those dependencies change then Accumulo might also need to change.

      HADOOP-13660 bumped commons-configuration from 1.6 -> 2.1 for Hadoop 3.0.0 and so NoClassDefFoundErrors are thrown per the below stack trace on accumulo init:

      [mike@localhost cloud]$ accumulo init
      2017-03-22 15:38:34,106 [start.Main] ERROR: Uncaught exception
      java.util.ServiceConfigurationError: org.apache.accumulo.start.spi.KeywordExecutable: Provider org.apache.accumulo.proxy.Proxy could not be instantiated
          at java.util.ServiceLoader.fail(ServiceLoader.java:232)
          at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
          at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
          at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
          at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
          at org.apache.accumulo.start.Main.checkDuplicates(Main.java:223)
          at org.apache.accumulo.start.Main.getExecutables(Main.java:215)
          at org.apache.accumulo.start.Main.main(Main.java:78)
      Caused by: java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
          at java.lang.Class.getDeclaredConstructors0(Native Method)
          at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
          at java.lang.Class.getConstructor0(Class.java:3075)
          at java.lang.Class.newInstance(Class.java:412)
          at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
          ... 5 more
      Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
          at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          at org.apache.accumulo.start.classloader.AccumuloClassLoader$2.loadClass(AccumuloClassLoader.java:284)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          ... 10 more
      

      I worked around this by dropping the commons-configuration 1.6 jar on Accumulo's classpath, but this should either be provided by Accumulo now or Accumulo should be bumped to commons-configuration2 as well. Note that the latter change would cause problems for Hadoop 2 installations.

      There was actually one more thing that needed added to the accumulo-site.xml in order to get Accumulo to run in Hadoop 3:

      $HADOOP_PREFIX/share/hadoop/client/[^.].*.jar
      

      The reason for this is because Hdfs.java moved from the hadoop-hdfs jar to the hadoop-client-api jar.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ctubbsii Christopher Tubbs
                Reporter:
                m-hogue Michael Hogue
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 20m
                  2h 20m