Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20201

HBase must provide commons-cli-1.4 for mapreduce jobs with H3

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: mapreduce
    • Labels:
      None

      Description

      Been trying to get some pre-existing mapreduce tests working against HBase2.

      There's an inherent problem right now that hadoop-common depends on commons-cli-1.2 and HBase depends on commons-cli-1.4. This means that if you use $(hbase mapredcp) to submit a mapreduce job via hadoop jar, you'll get an error like:

      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/DefaultParser
          at org.apache.hadoop.hbase.util.AbstractHBaseTool.isHelpCommand(AbstractHBaseTool.java:165)
          at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:133)
          at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
          at org.apache.hadoop.hbase.util.AbstractHBaseTool.doStaticMain(AbstractHBaseTool.java:270)
          at hbase_it.App.main(App.java:85)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
      Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.DefaultParser
          at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          ... 11 more

      My guess is that in previous versions, we didn't have this conflict with Hadoop (we were on the same version). Now, we're not.

      I see two routes:

      1. We just alter the mapredcp to include our "correct" commons-cli-1.4 on the classpath and remind users to make use of the HADOOP_USER_CLASSPATH_FIRST environment variable
      2. We put commons-cli into our hbase-thirdparty and stop using it directly.

      The former is definitely quicker, but I'm guessing the latter would insulate us more nicely.

      Thoughts, Michael Stack, Sean Busbey, Mike Drob (and others who have done H3 work?)

        Attachments

        1. HBASE-20201.001.patch
          66 kB
          Josh Elser

        Issue Links

          Activity

            People

            • Assignee:
              elserj Josh Elser
              Reporter:
              romil.choksi Romil Choksi

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment