HBase
  1. HBase
  2. HBASE-5286

bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.92.1, 0.94.0
    • Component/s: scripts
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Here's the bit from bin/hbase that might need TLC now that Hadoop can be spotted in the wild in split-package configuration:

      #If avail, add Hadoop to the CLASSPATH and to the JAVA_LIBRARY_PATH
      if [ ! -z $HADOOP_HOME ]; then
        HADOOPCPPATH=""
        if [ -z $HADOOP_CONF_DIR ]; then
          HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_HOME}/conf")
        else
          HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" "${HADOOP_CONF_DIR}")
        fi
        if [ "`echo ${HADOOP_HOME}/hadoop-core*.jar`" != "${HADOOP_HOME}/hadoop-core*.jar" ] ; then
          HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-core*.jar | head -1`)
        else
          HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-common*.jar | head -1`)
          HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-hdfs*.jar | head -1`)
          HADOOPCPPATH=$(append_path "${HADOOPCPPATH}" `ls ${HADOOP_HOME}/hadoop-mapred*.jar | head -1`)
        fi
      

      There's a couple of issues with the above code:
      0. HADOOP_HOME is now deprecated in Hadoop 0.23
      1. the list of jar files added to the class-path should be revised
      2. we need to figure out a more robust way to get the jar files that are needed to the classpath (things like hadoop-mapred*.jar tend to match src/test jars as well)

      Better yet, it would be useful to look into whether we can transition HBase's bin/hbase onto using bin/hadoop as a launcher script instead of direct JAVA invocations (Pig, Hive, Sqoop and Mahout already do that)

      1. HBASE-5286.patch.txt
        4 kB
        Roman Shaposhnik

        Issue Links

          Activity

          Roman Shaposhnik created issue -
          Jimmy Xiang made changes -
          Field Original Value New Value
          Link This issue is related to HBASE-5221 [ HBASE-5221 ]
          Todd Lipcon made changes -
          Comment [ Integrated in HBase-TRUNK #2649 (See [https://builds.apache.org/job/HBase-TRUNK/2649/])
              HBASE-5286 Added pointer to Lars Hofhansl's blog describing his attempt at adding a '...column prefix delete marker'

          stack :
          Files :
          * /hbase/trunk/src/docbkx/book.xml
          ]
          Roman Shaposhnik made changes -
          Attachment HBASE-5286.patch.txt [ 12516930 ]
          stack made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.92.1 [ 12318551 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Roman Shaposhnik
              Reporter:
              Roman Shaposhnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development