[HBASE-20201] HBase must provide commons-cli-1.4 for mapreduce jobs with H3 - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Done
Affects Version/s: None
Fix Version/s: 2.0.0
Component/s: mapreduce
Labels:
None

Description

Been trying to get some pre-existing mapreduce tests working against HBase2.

There's an inherent problem right now that hadoop-common depends on commons-cli-1.2 and HBase depends on commons-cli-1.4. This means that if you use $(hbase mapredcp) to submit a mapreduce job via hadoop jar, you'll get an error like:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/DefaultParser
    at org.apache.hadoop.hbase.util.AbstractHBaseTool.isHelpCommand(AbstractHBaseTool.java:165)
    at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:133)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:270)
    at hbase_it.App.main(App.java:85)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.DefaultParser
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 11 more

My guess is that in previous versions, we didn't have this conflict with Hadoop (we were on the same version). Now, we're not.

I see two routes:

We just alter the mapredcp to include our "correct" commons-cli-1.4 on the classpath and remind users to make use of the HADOOP_USER_CLASSPATH_FIRST environment variable
We put commons-cli into our hbase-thirdparty and stop using it directly.

The former is definitely quicker, but I'm guessing the latter would insulate us more nicely.

Thoughts, Michael Stack, Sean Busbey, Mike Drob (and others who have done H3 work?)