HBase
  1. HBase
  2. HBASE-5221

bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.92.0
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Running against an 0.24.0-SNAPSHOT hadoop:
      ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-common*.jar: No such file or directory
      ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-hdfs*.jar: No such file or directory
      ls: cannot access /home/todd/ha-demo/hadoop-0.24.0-SNAPSHOT/hadoop-mapred*.jar: No such file or directory
      The jars are rooted deeper in the heirarchy.

      1. hbase-5221.txt
        1 kB
        Jimmy Xiang

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2656 (See https://builds.apache.org/job/HBase-TRUNK/2656/)
          HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout – REVERTED
          HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout

          stack :
          Files :

          • /hbase/trunk/bin/hbase

          stack :
          Files :

          • /hbase/trunk/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2656 (See https://builds.apache.org/job/HBase-TRUNK/2656/ ) HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout – REVERTED HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout stack : Files : /hbase/trunk/bin/hbase stack : Files : /hbase/trunk/bin/hbase
          Hide
          Roman Shaposhnik added a comment -

          @stack, NP – thanks for taking care of it on your end so quickly. I'll try to tackle HBASE-5286 tomorrow.

          @Jimmy,

          for the developer build you always have an option of unsetting the HADOOP_HOME variable and the ClassNotFound will go away. In fact, I think, for dev. builds it is desirable to unset HADOOP_HOME, since that way you can be sure that you're testing your HBase build with whatever dependencies are declared in the POM.

          Show
          Roman Shaposhnik added a comment - @stack, NP – thanks for taking care of it on your end so quickly. I'll try to tackle HBASE-5286 tomorrow. @Jimmy, for the developer build you always have an option of unsetting the HADOOP_HOME variable and the ClassNotFound will go away. In fact, I think, for dev. builds it is desirable to unset HADOOP_HOME, since that way you can be sure that you're testing your HBase build with whatever dependencies are declared in the POM.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #102 (See https://builds.apache.org/job/HBase-TRUNK-security/102/)
          HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout – REVERTED
          HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout

          stack :
          Files :

          • /hbase/trunk/bin/hbase

          stack :
          Files :

          • /hbase/trunk/bin/hbase
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #102 (See https://builds.apache.org/job/HBase-TRUNK-security/102/ ) HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout – REVERTED HBASE-5221 bin/hbase script doesn't look for Hadoop jars in the right place in trunk layout stack : Files : /hbase/trunk/bin/hbase stack : Files : /hbase/trunk/bin/hbase
          Hide
          Jimmy Xiang added a comment -

          Ok, let me close it as dup. Probably, I just use the fix for myself before HBASE-5286 is resolved.

          Show
          Jimmy Xiang added a comment - Ok, let me close it as dup. Probably, I just use the fix for myself before HBASE-5286 is resolved.
          Hide
          Jimmy Xiang added a comment -

          The problem is that when I run hbase shell, it complains those files are missing, and ClassNotFound org.apache.hadoop.util.PlatformName.

          We need to fix it.

          The script is already looking under HADOOP installation tree, just a wrong place.

          I don't think this fix will break anything. We can use this fix before HBASE-5286 is resolved.

          Show
          Jimmy Xiang added a comment - The problem is that when I run hbase shell, it complains those files are missing, and ClassNotFound org.apache.hadoop.util.PlatformName. We need to fix it. The script is already looking under HADOOP installation tree, just a wrong place. I don't think this fix will break anything. We can use this fix before HBASE-5286 is resolved.
          Hide
          stack added a comment -

          @Roman Thanks for catching this.

          Show
          stack added a comment - @Roman Thanks for catching this.
          Hide
          stack added a comment -

          I reverted the change Jimmy. Should we close as won't resolve or as dup of the issue that Roman refers to?

          Show
          stack added a comment - I reverted the change Jimmy. Should we close as won't resolve or as dup of the issue that Roman refers to?
          Hide
          Roman Shaposhnik added a comment -

          Guys, this fix will break certain HADOOP 0.23 layout. The correct fix for this one will have to come out of HBASE-5286

          It would be really nice if folks can chime in on the JIRA that I quoted.

          Also, I don't quite understand why this is a problem. For developer's builds HBase bundles everything under lib and thus has no reason to look under HADOOP installation tree. For pacakged deployments (Bigtop, CDH) the layout of Hadoop is different anyway, which means this fix will actually break things.

          Show
          Roman Shaposhnik added a comment - Guys, this fix will break certain HADOOP 0.23 layout. The correct fix for this one will have to come out of HBASE-5286 It would be really nice if folks can chime in on the JIRA that I quoted. Also, I don't quite understand why this is a problem. For developer's builds HBase bundles everything under lib and thus has no reason to look under HADOOP installation tree. For pacakged deployments (Bigtop, CDH) the layout of Hadoop is different anyway, which means this fix will actually break things.
          Hide
          stack added a comment -

          Committed to trunk (Should we put it in 0.92?) Thanks for the patch Jimmy.

          Show
          stack added a comment - Committed to trunk (Should we put it in 0.92?) Thanks for the patch Jimmy.
          Hide
          stack added a comment -

          No worries Jimmy. That'll do. Thanks. Let me commit.

          Show
          stack added a comment - No worries Jimmy. That'll do. Thanks. Let me commit.
          Hide
          Jimmy Xiang added a comment -

          Since 0.23, Hadoop re-organized the folder structure. They put the jars under each individual modules like hdfs, mapreduce, util and so on (under share/hadoop). The common one is under share/haddop/common.

          I am not very clear about the story behind either. Todd should know this much better.

          Show
          Jimmy Xiang added a comment - Since 0.23, Hadoop re-organized the folder structure. They put the jars under each individual modules like hdfs, mapreduce, util and so on (under share/hadoop). The common one is under share/haddop/common. I am not very clear about the story behind either. Todd should know this much better.
          Hide
          stack added a comment -

          So idea is to put hadoop jars under /share? Where does this notion come from? (Pardon my ignorance)

          Show
          stack added a comment - So idea is to put hadoop jars under /share? Where does this notion come from? (Pardon my ignorance)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12513737/hbase-5221.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestAtomicOperation
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.mapred.TestTableMapReduce
          org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
          org.apache.hadoop.hbase.TestZooKeeper

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/919//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/919//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/919//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513737/hbase-5221.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.TestZooKeeper Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/919//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/919//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/919//console This message is automatically generated.
          Hide
          Jimmy Xiang added a comment -

          This patch works for hadoop-0.23 and versions with the old folder structure.

          Show
          Jimmy Xiang added a comment - This patch works for hadoop-0.23 and versions with the old folder structure.

            People

            • Assignee:
              Jimmy Xiang
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development