[HBASE-20201] HBase must provide commons-cli-1.4 for mapreduce jobs with H3

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Done
• Fix Version/s: 2.0.0
• Component/s: mapreduce
• Assignee: Josh Elser
• Reporter: Romil Choksi
• Created: 14/Mar/18 18:44
• Updated: 01/Aug/18 06:22
• Resolved: 19/Mar/18 18:52

Description

Been trying to get some pre-existing mapreduce tests working against HBase2.

There's an inherent problem right now that hadoop-common depends on commons-cli-1.2 and HBase depends on commons-cli-1.4. This means that if you use $(hbase mapredcp) to submit a mapreduce job via hadoop jar, you'll get an error like:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/DefaultParser
    at org.apache.hadoop.hbase.util.AbstractHBaseTool.isHelpCommand(AbstractHBaseTool.java:165)
    at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:133)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:270)
    at hbase_it.App.main(App.java:85)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.DefaultParser
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 11 more

My guess is that in previous versions, we didn't have this conflict with Hadoop (we were on the same version). Now, we're not.

I see two routes:

1. We just alter the mapredcp to include our "correct" commons-cli-1.4 on the classpath and remind users to make use of the HADOOP_USER_CLASSPATH_FIRST environment variable
2. We put commons-cli into our hbase-thirdparty and stop using it directly.

The former is definitely quicker, but I'm guessing the latter would insulate us more nicely.

Thoughts, stack, busbey, mdrob (and others who have done H3 work?)

Issue Links

• is blocked by: HBASE-20216 [thirdparty] Bundle commons-cli
• is caused by: HBASE-17050 Upgrade Apache CLI version from 1.2 to 1.3.1
• is superceded by: HBASE-20223 Use hbase-thirdparty 2.1.0