HBase
  1. HBase
  2. HBASE-4467

Handle inconsistencies in Hadoop libraries naming in hbase script

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Not a Problem
    • Affects Version/s: 0.92.0, 0.94.0
    • Fix Version/s: None
    • Component/s: scripts
    • Labels:
      None

      Description

      When using an Hadoop tarball that has a library naming of "hadoop-x.y.z-core" as opposed to "hadoop-core-x.y.z" then the hbase script throws errors.

      $ bin/start-hbase.sh 
      ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file or directory
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file or directory
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      localhost: starting zookeeper, logging to /projects/opensource/hbase-trunk-rw//logs/hbase-larsgeorge-zookeeper-de1-app-mbp-2.out
      localhost: /projects/opensource/hadoop-0.20.2-append
      localhost: ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file or directory
      localhost: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
      localhost: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
      localhost: 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      localhost: 	at java.security.AccessController.doPrivileged(Native Method)
      localhost: 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      localhost: 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      localhost: 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      localhost: 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      starting master, logging to /projects/opensource/hbase-trunk-rw/bin/../logs/hbase-larsgeorge-master-de1-app-mbp-2.out
      /projects/opensource/hadoop-0.20.2-append
      ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file or directory
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      localhost: starting regionserver, logging to /projects/opensource/hbase-trunk-rw//logs/hbase-larsgeorge-regionserver-de1-app-mbp-2.out
      localhost: /projects/opensource/hadoop-0.20.2-append
      localhost: ls: /projects/opensource/hadoop-0.20.2-append/hadoop-core*.jar: No such file or directory
      localhost: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
      localhost: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
      localhost: 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      localhost: 	at java.security.AccessController.doPrivileged(Native Method)
      localhost: 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      localhost: 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      localhost: 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      localhost: 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      

      The naming in this case:

      $ ll /projects/opensource/hadoop-0.20.2-append/
      total 14960
      drwxr-xr-x@  26 larsgeorge  staff      884 Apr 13 09:09 .
      drwxr-xr-x  114 larsgeorge  staff     3876 Sep 22 19:42 ..
      -rw-r--r--@   1 larsgeorge  staff   348624 Apr 13 08:58 CHANGES.txt
      -rw-r--r--@   1 larsgeorge  staff    13366 Apr 13 08:58 LICENSE.txt
      -rw-r--r--@   1 larsgeorge  staff      101 Apr 13 08:58 NOTICE.txt
      -rw-r--r--@   1 larsgeorge  staff     1366 Apr 13 08:58 README.txt
      drwxr-xr-x@  17 larsgeorge  staff      578 Apr 13 08:58 bin
      -rw-r--r--@   1 larsgeorge  staff    74035 Apr 13 08:58 build.xml
      drwxr-xr-x@   4 larsgeorge  staff      136 Apr 13 08:58 c++
      drwxr-xr-x   18 larsgeorge  staff      612 Aug  9 15:11 conf
      drwxr-xr-x@  15 larsgeorge  staff      510 Apr 13 08:58 conf.original
      drwxr-xr-x@  13 larsgeorge  staff      442 Apr 13 08:58 contrib
      drwxr-xr-x@  63 larsgeorge  staff     2142 Apr 13 08:58 docs
      -rw-r--r--@   1 larsgeorge  staff     6839 Apr 13 08:58 hadoop-0.20.2-ant.jar
      -rw-r--r--    1 larsgeorge  staff  2707920 Apr 13 09:06 hadoop-0.20.2-core.jar
      -rw-r--r--@   1 larsgeorge  staff  2689741 Apr 13 08:58 hadoop-0.20.2-core.jar.original
      -rw-r--r--@   1 larsgeorge  staff   142466 Apr 13 08:58 hadoop-0.20.2-examples.jar
      -rw-r--r--@   1 larsgeorge  staff  1563859 Apr 13 08:58 hadoop-0.20.2-test.jar
      -rw-r--r--@   1 larsgeorge  staff    69940 Apr 13 08:58 hadoop-0.20.2-tools.jar
      drwxr-xr-x@   6 larsgeorge  staff      204 Apr 13 08:58 ivy
      -rw-r--r--@   1 larsgeorge  staff     8852 Apr 13 08:58 ivy.xml
      drwxr-xr-x@  30 larsgeorge  staff     1020 Jul 13 10:20 lib
      drwxr-xr-x@   3 larsgeorge  staff      102 Apr 13 08:58 librecordio
      drwxr-xr-x    3 larsgeorge  staff      102 May 16 09:56 logs
      drwxr-xr-x@  17 larsgeorge  staff      578 Apr 13 08:58 src
      drwxr-xr-x@   8 larsgeorge  staff      272 Apr 13 08:58 webapps
      
      1. HBASE-4467.patch
        0.5 kB
        Lars George

        Activity

        Hide
        Lars George added a comment -

        Hmm, even changing the "ls" to use only "hadoop-*.jar" does not fix this. It seems that the path is constructed wrong?

        localhost: classpath: /projects/opensource/hadoop-0.20.2-append/conf:/projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-ant.jar
        localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-core.jar
        localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-examples.jar
        localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-test.jar
        localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-tools.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-cli-1.2.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-codec-1.3.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-el-1.0.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-httpclient-3.0.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-logging-1.0.4.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-logging-api-1.0.4.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-net-1.4.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/core-3.1.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/hsqldb-1.8.0.10.jar:/projects/opensource/hadoop-0.20.2-append/lib/jasper-compiler-5.5.12.jar:/projects/opensource/hadoop-0.20.2-append/lib/jasper-runtime-5.5.12.jar:/projects/opensource/hadoop-0.20.2-append/lib/jets3t-0.6.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/jetty-6.1.14.jar:/projects/opensource/hadoop-0.20.2-append/lib/jetty-util-6.1.14.jar:/projects/opensource/hadoop-0.20.2-append/lib/junit-3.8.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/kfs-0.2.2.jar:/projects/opensource/hadoop-0.20.2-append/lib/log4j-1.2.15.jar:/projects/opensource/hadoop-0.20.2-append/lib/mockito-all-1.8.0.jar:/projects/opensource/hadoop-0.20.2-append/lib/oro-2.0.8.jar:/projects/opensource/hadoop-0.20.2-append/lib/servlet-api-2.5-6.1.14.jar:/projects/opensource/hadoop-0.20.2-append/lib/slf4j-api-1.4.3.jar:/projects/opensource/hadoop-0.20.2-append/lib/slf4j-log4j12-1.4.3.jar:/projects/opensource/hadoop-0.20.2-append/lib/xmlenc-0.52.jar
        localhost: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
        localhost: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
        localhost: 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        localhost: 	at java.security.AccessController.doPrivileged(Native Method)
        

        It seems to add newlines into the classpath? Weird.

        Show
        Lars George added a comment - Hmm, even changing the "ls" to use only "hadoop-*.jar" does not fix this. It seems that the path is constructed wrong? localhost: classpath: /projects/opensource/hadoop-0.20.2-append/conf:/projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-ant.jar localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-core.jar localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-examples.jar localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-test.jar localhost: /projects/opensource/hadoop-0.20.2-append/hadoop-0.20.2-tools.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-cli-1.2.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-codec-1.3.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-el-1.0.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-httpclient-3.0.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-logging-1.0.4.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-logging-api-1.0.4.jar:/projects/opensource/hadoop-0.20.2-append/lib/commons-net-1.4.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/core-3.1.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/hsqldb-1.8.0.10.jar:/projects/opensource/hadoop-0.20.2-append/lib/jasper-compiler-5.5.12.jar:/projects/opensource/hadoop-0.20.2-append/lib/jasper-runtime-5.5.12.jar:/projects/opensource/hadoop-0.20.2-append/lib/jets3t-0.6.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/jetty-6.1.14.jar:/projects/opensource/hadoop-0.20.2-append/lib/jetty-util-6.1.14.jar:/projects/opensource/hadoop-0.20.2-append/lib/junit-3.8.1.jar:/projects/opensource/hadoop-0.20.2-append/lib/kfs-0.2.2.jar:/projects/opensource/hadoop-0.20.2-append/lib/log4j-1.2.15.jar:/projects/opensource/hadoop-0.20.2-append/lib/mockito-all-1.8.0.jar:/projects/opensource/hadoop-0.20.2-append/lib/oro-2.0.8.jar:/projects/opensource/hadoop-0.20.2-append/lib/servlet-api-2.5-6.1.14.jar:/projects/opensource/hadoop-0.20.2-append/lib/slf4j-api-1.4.3.jar:/projects/opensource/hadoop-0.20.2-append/lib/slf4j-log4j12-1.4.3.jar:/projects/opensource/hadoop-0.20.2-append/lib/xmlenc-0.52.jar localhost: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName localhost: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName localhost: at java.net.URLClassLoader$1.run(URLClassLoader.java:202) localhost: at java.security.AccessController.doPrivileged(Native Method) It seems to add newlines into the classpath? Weird.
        Hide
        Lars George added a comment - - edited

        So the issue is this:

        function append_path() {
          if [ -z "$1" ]; then
            echo $2
          else
            echo $1:$2
          fi
        }
        

        in combination with

         HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-*.jar`)
        

        This "ls" emits multiple values and that screws up the classpath. This either needs a "sed" or a loop outside of the "append_path" call iterating over all dir entries and adding them one by one.

        PS: Sorry for the edit, my browser (Chrome) is going bonkers with JIRA right now

        Show
        Lars George added a comment - - edited So the issue is this: function append_path() { if [ -z "$1" ]; then echo $2 else echo $1:$2 fi } in combination with HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-*.jar`) This "ls" emits multiple values and that screws up the classpath. This either needs a "sed" or a loop outside of the "append_path" call iterating over all dir entries and adding them one by one. PS: Sorry for the edit, my browser (Chrome) is going bonkers with JIRA right now
        Hide
        Lars George added a comment -

        Patch fixes creation of HADOOPCPPATH.

        Show
        Lars George added a comment - Patch fixes creation of HADOOPCPPATH.
        Hide
        stack added a comment -

        +1 on patch. If it works for you commit LarsG.

        Show
        stack added a comment - +1 on patch. If it works for you commit LarsG.
        Hide
        stack added a comment -

        @LarsG You going to commit?

        Show
        stack added a comment - @LarsG You going to commit?
        Hide
        stack added a comment -

        @LarsG You going to commit boss?

        Show
        stack added a comment - @LarsG You going to commit boss?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12496268/HBASE-4467.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/179//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12496268/HBASE-4467.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/179//console This message is automatically generated.
        Hide
        Jesse Yates added a comment -

        Can we remove 0.94 from this (if not close)? Looks like the issue is a non-starter there.

        Show
        Jesse Yates added a comment - Can we remove 0.94 from this (if not close)? Looks like the issue is a non-starter there.
        Hide
        Harsh J added a comment -

        I think this issue has gone stale, so am marking it resolved.

        • We pull in hadoop classpath if we find it.
        • We have a strict minimum version pinned today (1.x) which hasn't the renamed jars issue.
        Show
        Harsh J added a comment - I think this issue has gone stale, so am marking it resolved. We pull in hadoop classpath if we find it. We have a strict minimum version pinned today (1.x) which hasn't the renamed jars issue.

          People

          • Assignee:
            Lars George
            Reporter:
            Lars George
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development