Been trying to get some pre-existing mapreduce tests working against HBase2.
There's an inherent problem right now that hadoop-common depends on commons-cli-1.2 and HBase depends on commons-cli-1.4. This means that if you use $(hbase mapredcp) to submit a mapreduce job via hadoop jar, you'll get an error like:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/DefaultParser at org.apache.hadoop.hbase.util.AbstractHBaseTool.isHelpCommand(AbstractHBaseTool.java:165) at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:133) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:270) at hbase_it.App.main(App.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:239) at org.apache.hadoop.util.RunJar.main(RunJar.java:153) Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.DefaultParser at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 11 more
My guess is that in previous versions, we didn't have this conflict with Hadoop (we were on the same version). Now, we're not.
I see two routes:
- We just alter the mapredcp to include our "correct" commons-cli-1.4 on the classpath and remind users to make use of the HADOOP_USER_CLASSPATH_FIRST environment variable
- We put commons-cli into our hbase-thirdparty and stop using it directly.
The former is definitely quicker, but I'm guessing the latter would insulate us more nicely.
Thoughts, Michael Stack, Sean Busbey, Mike Drob (and others who have done H3 work?)
- is blocked by
-
HBASE-20216 [thirdparty] Bundle commons-cli
-
- Resolved
-
- is caused by
-
HBASE-17050 Upgrade Apache CLI version from 1.2 to 1.3.1
-
- Resolved
-
- is superceded by
-
HBASE-20223 Use hbase-thirdparty 2.1.0
-
- Resolved
-