Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-13120

Allow disabling hadoop classpath and native library lookup

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.0
    • Fix Version/s: 2.0.0, 1.0.1, 1.1.0, 0.98.12
    • Component/s: hbase
    • Labels:
      None

      Description

      • Current bin/hbase script sets the java.library.path to include hadoop native libs based on what version of hadoop is installed on the box.
        HADOOP_IN_PATH=$(PATH="${HADOOP_HOME:-${HADOOP_PREFIX}}/bin:$PATH" which hadoop 2>/dev/null)
        
      • This effectively means that a self-contained HBase running with a different version of embedded hadoop jars will fail to work in case of version incompatibilities.
      • Example: AMBARI-5707, runs a embedded HBase on local FS side-by-side with cluster Hadoop installation.
      • It would be good to have a hbase-env variable to completely override native lib path or a config to disable native lib path lookup, in which case user has to provide it during start.
      1. HBASE-13120-1.patch
        2 kB
        Siddharth Wagle
      2. HBASE-13120.patch
        2 kB
        Siddharth Wagle

        Activity

        Hide
        swagle Siddharth Wagle added a comment -

        Verified manually that patch works.

        Show
        swagle Siddharth Wagle added a comment - Verified manually that patch works.
        Hide
        enis Enis Soztutar added a comment -

        +1. I think this is a valid use case where there is hadoop installed on the node(s), but we are trying to run HBase from a self-contained tarball. We should not try to find the "installed" hadoop in this case.

        Show
        enis Enis Soztutar added a comment - +1. I think this is a valid use case where there is hadoop installed on the node(s), but we are trying to run HBase from a self-contained tarball. We should not try to find the "installed" hadoop in this case.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12701235/HBASE-13120.patch
        against master branch at commit 49b4f3737eb0dd7c5d88d6dcbe8b5d4f167c6a2b.
        ATTACHMENT ID: 12701235

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.
        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701235/HBASE-13120.patch against master branch at commit 49b4f3737eb0dd7c5d88d6dcbe8b5d4f167c6a2b. ATTACHMENT ID: 12701235 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12989//console This message is automatically generated.
        Hide
        busbey Sean Busbey added a comment -

        +if [ "$DISABLE_HADOOP_CLASSPATH_LOOKUP" != "true" ] ; then

        Please name this HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP. Not that I think we're likely to conflict on the name, but just in case.

        Also, if you just check if the string is non-zero length it'll be easier to integrate programmatically determining if we should disable.

        I think this is a valid use case where there is hadoop installed on the node(s), but we are trying to run HBase from a self-contained tarball. We should not try to find the "installed" hadoop in this case.

        This is a very good point. How much more work would it be to get this exported when we're starting local distributed mode? At a minimum we should add a note to the ref guide telling folks to set it.

        Show
        busbey Sean Busbey added a comment - +if [ "$DISABLE_HADOOP_CLASSPATH_LOOKUP" != "true" ] ; then Please name this HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP. Not that I think we're likely to conflict on the name, but just in case. Also, if you just check if the string is non-zero length it'll be easier to integrate programmatically determining if we should disable. I think this is a valid use case where there is hadoop installed on the node(s), but we are trying to run HBase from a self-contained tarball. We should not try to find the "installed" hadoop in this case. This is a very good point. How much more work would it be to get this exported when we're starting local distributed mode? At a minimum we should add a note to the ref guide telling folks to set it.
        Hide
        swagle Siddharth Wagle added a comment -

        Sean Busbey Renamed.

        if you just check if the string is non-zero length it'll be easier to integrate programmatically determining if we should disable.
        I initially did this but then changed to explicit check for true in order to leave only a single case for disable and regular behavior for all other cases, for example someone doing a HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP="false" and the lookup still getting disabled.

        Show
        swagle Siddharth Wagle added a comment - Sean Busbey Renamed. if you just check if the string is non-zero length it'll be easier to integrate programmatically determining if we should disable. I initially did this but then changed to explicit check for true in order to leave only a single case for disable and regular behavior for all other cases, for example someone doing a HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP="false" and the lookup still getting disabled.
        Hide
        busbey Sean Busbey added a comment -

        wfm.

        Show
        busbey Sean Busbey added a comment - wfm.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12701417/HBASE-13120-1.patch
        against master branch at commit d1619bceb3142f3ab8c134365e18a150fbd5b9bf.
        ATTACHMENT ID: 12701417

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.
        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0)

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 checkstyle. The applied patch does not increase the total number of checkstyle errors

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.wal.TestWALReplay

        -1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver.testCoprocessorHooksInRegionsMerge(TestRegionServerObserver.java:88)
        at org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint.testCoprocessorError(TestCoprocessorEndpoint.java:308)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12701417/HBASE-13120-1.patch against master branch at commit d1619bceb3142f3ab8c134365e18a150fbd5b9bf. ATTACHMENT ID: 12701417 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.1 2.5.2 2.6.0) +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 checkstyle . The applied patch does not increase the total number of checkstyle errors +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestWALReplay -1 core zombie tests . There are 2 zombie test(s): at org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver.testCoprocessorHooksInRegionsMerge(TestRegionServerObserver.java:88) at org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint.testCoprocessorError(TestCoprocessorEndpoint.java:308) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/13002//console This message is automatically generated.
        Hide
        enis Enis Soztutar added a comment -

        I have committed this. Thanks Sid for the patch.

        Show
        enis Enis Soztutar added a comment - I have committed this. Thanks Sid for the patch.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-TRUNK #6180 (See https://builds.apache.org/job/HBase-TRUNK/6180/)
        HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (enis: rev ec877959d7611ed8edfb15d67f5a662080c2e6da)

        • bin/hbase
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #6180 (See https://builds.apache.org/job/HBase-TRUNK/6180/ ) HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (enis: rev ec877959d7611ed8edfb15d67f5a662080c2e6da) bin/hbase
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-1.1 #223 (See https://builds.apache.org/job/HBase-1.1/223/)
        HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (enis: rev 287d08447f5de0fd2d7d5c8bc15d7221b1be8c5e)

        • bin/hbase
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-1.1 #223 (See https://builds.apache.org/job/HBase-1.1/223/ ) HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (enis: rev 287d08447f5de0fd2d7d5c8bc15d7221b1be8c5e) bin/hbase
        Hide
        apurtell Andrew Purtell added a comment -

        This has an affects version of 0.98. Let me look at this change for there too.

        Show
        apurtell Andrew Purtell added a comment - This has an affects version of 0.98. Let me look at this change for there too.
        Hide
        enis Enis Soztutar added a comment -

        Andrew Purtell do you mind committing this to 1.0 as well if you want to pick it in 0.98. It is a low risk change.

        Show
        enis Enis Soztutar added a comment - Andrew Purtell do you mind committing this to 1.0 as well if you want to pick it in 0.98. It is a low risk change.
        Hide
        apurtell Andrew Purtell added a comment -
        Show
        apurtell Andrew Purtell added a comment - Sure Enis Soztutar
        Hide
        apurtell Andrew Purtell added a comment -

        Pushed to 0.98 and branch-1.0

        Show
        apurtell Andrew Purtell added a comment - Pushed to 0.98 and branch-1.0
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-1.0 #787 (See https://builds.apache.org/job/HBase-1.0/787/)
        HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (apurtell: rev acef2a77f8c2c65912956de3e0d040f5059f7f92)

        • bin/hbase
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-1.0 #787 (See https://builds.apache.org/job/HBase-1.0/787/ ) HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (apurtell: rev acef2a77f8c2c65912956de3e0d040f5059f7f92) bin/hbase
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in HBase-0.98 #882 (See https://builds.apache.org/job/HBase-0.98/882/)
        HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (apurtell: rev 2aa75a6175e36539e105d8dcbdf640af072f322e)

        • bin/hbase
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #882 (See https://builds.apache.org/job/HBase-0.98/882/ ) HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (apurtell: rev 2aa75a6175e36539e105d8dcbdf640af072f322e) bin/hbase
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #839 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/839/)
        HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (apurtell: rev 2aa75a6175e36539e105d8dcbdf640af072f322e)

        • bin/hbase
        Show
        hudson Hudson added a comment - FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #839 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/839/ ) HBASE-13120 Allow disabling hadoop classpath and native library lookup (Siddharth Wagle) (apurtell: rev 2aa75a6175e36539e105d8dcbdf640af072f322e) bin/hbase
        Hide
        enis Enis Soztutar added a comment -

        Closing this issue after 1.0.1 release.

        Show
        enis Enis Soztutar added a comment - Closing this issue after 1.0.1 release.

          People

          • Assignee:
            Unassigned
            Reporter:
            swagle Siddharth Wagle
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development