[ACCUMULO-4611] Provide a migration path to commons-configuration2 to better support Hadoop 3

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.7.2, 1.8.1
• Fix Version/s: 1.9.0
• Labels: pull-request-available

Environment

CentOS 7
Accumulo 1.7.x, 1.8.x
Hadoop 3.0.0-alpha 2
Zookeeper 3.4.9

• Assignee: Christopher Tubbs
• Reporter: Michael Hogue
• Created: 22/Mar/17 19:48
• Updated: 23/Apr/19 16:12
• Resolved: 22/Feb/18 23:42

Description

I was investigating running either Accumulo 1.7.x or 1.8.x on Hadoop 3.0.0-alpha 2 and ran into a couple of issues. Since Accumulo assumes some dependencies will be provided by Hadoop (per ACCUMULO-1244), if those dependencies change then Accumulo might also need to change.

HADOOP-13660 bumped commons-configuration from 1.6 -> 2.1 for Hadoop 3.0.0 and so NoClassDefFoundErrors are thrown per the below stack trace on accumulo init:

[mike@localhost cloud]$ accumulo init
2017-03-22 15:38:34,106 [start.Main] ERROR: Uncaught exception
java.util.ServiceConfigurationError: org.apache.accumulo.start.spi.KeywordExecutable: Provider org.apache.accumulo.proxy.Proxy could not be instantiated
    at java.util.ServiceLoader.fail(ServiceLoader.java:232)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.accumulo.start.Main.checkDuplicates(Main.java:223)
    at org.apache.accumulo.start.Main.getExecutables(Main.java:215)
    at org.apache.accumulo.start.Main.main(Main.java:78)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
    at java.lang.Class.getDeclaredConstructors0(Native Method)
    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
    at java.lang.Class.getConstructor0(Class.java:3075)
    at java.lang.Class.newInstance(Class.java:412)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
    ... 5 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at org.apache.accumulo.start.classloader.AccumuloClassLoader$2.loadClass(AccumuloClassLoader.java:284)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 10 more

I worked around this by dropping the commons-configuration 1.6 jar on Accumulo's classpath, but this should either be provided by Accumulo now or Accumulo should be bumped to commons-configuration2 as well. Note that the latter change would cause problems for Hadoop 2 installations.

There was actually one more thing that needed added to the accumulo-site.xml in order to get Accumulo to run in Hadoop 3:

$HADOOP_PREFIX/share/hadoop/client/[^.].*.jar

The reason for this is because Hdfs.java moved from the hadoop-hdfs jar to the hadoop-client-api jar.

Issue Links

• is related to: ACCUMULO-4753 Update to commons-configuration2
• relates to: HADOOP-13660 Upgrade commons-configuration version to 2.1
• links to: GitHub Pull Request #357
• links to: GitHub Pull Request #357