HBase
  1. HBase
  2. HBASE-5684

Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.

      1. ASF.LICENSE.NOT.GRANTED--D2709.1.patch
        40 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--D2709.2.patch
        41 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--D2709.3.patch
        40 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--D2709.4.patch
        40 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--D2757.1.patch
        45 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--D2757.2.patch
        45 kB
        Phabricator
      7. jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
        50 kB
        Mikhail Bautin
      8. ASF.LICENSE.NOT.GRANTED--D2757.3.patch
        44 kB
        Phabricator
      9. ASF.LICENSE.NOT.GRANTED--D2757.4.patch
        44 kB
        Phabricator

        Activity

        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #171 (See https://builds.apache.org/job/HBase-TRUNK-security/171/)
        [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more
        robust

        Summary:
        Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We
        need it to start a process-based HDFS cluster as well. We also need to make the
        whole thing more stable so we can use it in unit tests.

        Also all logs of local HBase cluster daemons are now tailed to the primary log
        of the unit test with the appropriate prefixes to make debugging easier.

        This is a trunk diff. The 89-fb version is D2709.

        Test Plan:
        Run the new unit test multiple times (10x or 50x).
        Run all unit tests.

        Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA

        Reviewed By: tedyu

        Differential Revision: https://reviews.facebook.net/D2757 (Revision 1326036)

        Result = FAILURE
        mbautin :
        Files :

        • /hbase/trunk/bin/test
        • /hbase/trunk/bin/test/process_based_cluster.sh
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HTestConst.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #171 (See https://builds.apache.org/job/HBase-TRUNK-security/171/ ) [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust Summary: Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier. This is a trunk diff. The 89-fb version is D2709. Test Plan: Run the new unit test multiple times (10x or 50x). Run all unit tests. Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA Reviewed By: tedyu Differential Revision: https://reviews.facebook.net/D2757 (Revision 1326036) Result = FAILURE mbautin : Files : /hbase/trunk/bin/test /hbase/trunk/bin/test/process_based_cluster.sh /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HTestConst.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Phabricator added a comment -

        mbautin has committed the revision "[jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        COMMIT
        https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1326045

        Show
        Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". REVISION DETAIL https://reviews.facebook.net/D2709 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1326045
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2759 (See https://builds.apache.org/job/HBase-TRUNK/2759/)
        [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more
        robust

        Summary:
        Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We
        need it to start a process-based HDFS cluster as well. We also need to make the
        whole thing more stable so we can use it in unit tests.

        Also all logs of local HBase cluster daemons are now tailed to the primary log
        of the unit test with the appropriate prefixes to make debugging easier.

        This is a trunk diff. The 89-fb version is D2709.

        Test Plan:
        Run the new unit test multiple times (10x or 50x).
        Run all unit tests.

        Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA

        Reviewed By: tedyu

        Differential Revision: https://reviews.facebook.net/D2757 (Revision 1326036)

        Result = FAILURE
        mbautin :
        Files :

        • /hbase/trunk/bin/test
        • /hbase/trunk/bin/test/process_based_cluster.sh
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
        • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HTestConst.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2759 (See https://builds.apache.org/job/HBase-TRUNK/2759/ ) [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust Summary: Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier. This is a trunk diff. The 89-fb version is D2709. Test Plan: Run the new unit test multiple times (10x or 50x). Run all unit tests. Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA Reviewed By: tedyu Differential Revision: https://reviews.facebook.net/D2757 (Revision 1326036) Result = FAILURE mbautin : Files : /hbase/trunk/bin/test /hbase/trunk/bin/test/process_based_cluster.sh /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HTestConst.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Phabricator added a comment -

        mbautin has committed the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        COMMIT
        https://reviews.facebook.net/rHBASE1326036

        Show
        Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". REVISION DETAIL https://reviews.facebook.net/D2757 COMMIT https://reviews.facebook.net/rHBASE1326036
        Hide
        Mikhail Bautin added a comment -

        Committed to trunk.

        Show
        Mikhail Bautin added a comment - Committed to trunk.
        Hide
        Phabricator added a comment -

        mbautin updated the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA

        Fixing a couple more minor things.

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        AFFECTED FILES
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        Show
        Phabricator added a comment - mbautin updated the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA Fixing a couple more minor things. REVISION DETAIL https://reviews.facebook.net/D2757 AFFECTED FILES bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522636/D2757.3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 29 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.replication.TestReplication

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522636/D2757.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 29 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1519//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        mbautin updated the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA

        Rebasing on recent changes in trunk.

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        AFFECTED FILES
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        Show
        Phabricator added a comment - mbautin updated the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA Rebasing on recent changes in trunk. REVISION DETAIL https://reviews.facebook.net/D2757 AFFECTED FILES bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522525/jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 40 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.replication.TestReplication
        org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
        org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
        org.apache.hadoop.hbase.regionserver.wal.TestHLog
        org.apache.hadoop.hbase.replication.TestMasterReplication

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522525/jira-HBASE-5684-Make-ProcessBasedLocalHBaseCluster-r-2012-04-12_20_42_02.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 40 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.regionserver.wal.TestHLog org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1506//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522524/D2757.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 30 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1505//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522524/D2757.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 30 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1505//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        mbautin updated the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA

        Setting ZooKeeper data directory and adding test category.

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        AFFECTED FILES
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        Show
        Phabricator added a comment - mbautin updated the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA Setting ZooKeeper data directory and adding test category. REVISION DETAIL https://reviews.facebook.net/D2757 AFFECTED FILES bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        INLINE COMMENTS
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 This is actually a stability fix. We have seen the load balancer shedding regions from most loaded regionservers in some tests.

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        BRANCH
        make_processbasedlocalhbasecluster_run_HBASE-5684_v11

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 This is actually a stability fix. We have seen the load balancer shedding regions from most loaded regionservers in some tests. REVISION DETAIL https://reviews.facebook.net/D2757 BRANCH make_processbasedlocalhbasecluster_run_ HBASE-5684 _v11
        Hide
        Phabricator added a comment -

        tedyu has accepted the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        INLINE COMMENTS
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 Is this restriction needed ?

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        BRANCH
        make_processbasedlocalhbasecluster_run_HBASE-5684_v11

        Show
        Phabricator added a comment - tedyu has accepted the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java:93 Is this restriction needed ? REVISION DETAIL https://reviews.facebook.net/D2757 BRANCH make_processbasedlocalhbasecluster_run_ HBASE-5684 _v11
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12522355/D2757.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 30 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1489//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522355/D2757.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 30 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1489//console This message is automatically generated.
        Hide
        Phabricator added a comment -

        mbautin requested code review of "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA

        Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.

        Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier.

        This is a trunk diff. The 89-fb version is D2709.

        TEST PLAN
        Run the new unit test multiple times (10x or 50x).
        Run all unit tests.

        REVISION DETAIL
        https://reviews.facebook.net/D2757

        AFFECTED FILES
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/6285/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - mbautin requested code review of " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: tedyu, stack, lhofhansl, nspiegelberg, amirshim, JIRA Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier. This is a trunk diff. The 89-fb version is D2709. TEST PLAN Run the new unit test multiple times (10x or 50x). Run all unit tests. REVISION DETAIL https://reviews.facebook.net/D2757 AFFECTED FILES bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6285/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Phabricator added a comment -

        amirshim has commented on the revision "[jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        Looks good.

        As we talked, maybe we should use [[http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html | WatchService]] for log tailer once we decide to move to Java 7.

        And we might want to split HBaseTestingUtility.java at some point since it's over 1500 lines.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        BRANCH
        make_processbasedlocalhbasecluster_run_HBASE-5684_v10

        Show
        Phabricator added a comment - amirshim has commented on the revision " [jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Looks good. As we talked, maybe we should use [ [http://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html | WatchService] ] for log tailer once we decide to move to Java 7. And we might want to split HBaseTestingUtility.java at some point since it's over 1500 lines. REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_ HBASE-5684 _v10
        Hide
        Phabricator added a comment -

        Liyin has commented on the revision "[jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        Mikhail, nice work LGTM !

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        BRANCH
        make_processbasedlocalhbasecluster_run_HBASE-5684_v10

        Show
        Phabricator added a comment - Liyin has commented on the revision " [jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Mikhail, nice work LGTM ! REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_ HBASE-5684 _v10
        Hide
        Phabricator added a comment -

        mbautin updated the revision "[jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA, tedyu

        Setting test timeout.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        AFFECTED FILES
        bin/hbase-config.sh
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        Show
        Phabricator added a comment - mbautin updated the revision " [jira] HBASE-5684 [89-fb] Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA, tedyu Setting test timeout. REVISION DETAIL https://reviews.facebook.net/D2709 AFFECTED FILES bin/hbase-config.sh bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Ted Yu added a comment -

        do you mean "Unit Tests Skipped"

        Yes.

        Show
        Ted Yu added a comment - do you mean "Unit Tests Skipped" Yes.
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        Ted: do you mean "Unit Tests Skipped" at the top of the diff? We currently don't have integration between HBase unit tests and Arcanist/Phabricator.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        BRANCH
        make_processbasedlocalhbasecluster_run_HBASE-5684_v9

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Ted: do you mean "Unit Tests Skipped" at the top of the diff? We currently don't have integration between HBase unit tests and Arcanist/Phabricator. REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_ HBASE-5684 _v9
        Hide
        Phabricator added a comment -

        tedyu has accepted the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        I still see 'This diff has Unit Test Problems.'

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        BRANCH
        make_processbasedlocalhbasecluster_run_HBASE-5684_v9

        Show
        Phabricator added a comment - tedyu has accepted the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". I still see 'This diff has Unit Test Problems.' REVISION DETAIL https://reviews.facebook.net/D2709 BRANCH make_processbasedlocalhbasecluster_run_ HBASE-5684 _v9
        Hide
        Phabricator added a comment -

        mbautin updated the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA

        Ted: I have moved testHomePath to the other unit test.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        AFFECTED FILES
        bin/hbase-config.sh
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        Show
        Phabricator added a comment - mbautin updated the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA Ted: I have moved testHomePath to the other unit test. REVISION DETAIL https://reviews.facebook.net/D2709 AFFECTED FILES bin/hbase-config.sh bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Phabricator added a comment -

        tedyu has commented on the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        Patch looks good.
        I was warned that there was unit test problem when I tried to accept.

        INLINE COMMENTS
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java:24 This test doesn't have assertion.
        Is it possible to merge it into other test classes ?

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        Show
        Phabricator added a comment - tedyu has commented on the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Patch looks good. I was warned that there was unit test problem when I tried to accept. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java:24 This test doesn't have assertion. Is it possible to merge it into other test classes ? REVISION DETAIL https://reviews.facebook.net/D2709
        Hide
        Phabricator added a comment -

        mbautin updated the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA

        Addressing Ted's comments. Also improving the utility script to be able to both show and kill local cluster daemons, now using the pid files stored in target/local_cluster.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        AFFECTED FILES
        bin/hbase-config.sh
        bin/test/process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        Show
        Phabricator added a comment - mbautin updated the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA Addressing Ted's comments. Also improving the utility script to be able to both show and kill local cluster daemons, now using the pid files stored in target/local_cluster. REVISION DETAIL https://reviews.facebook.net/D2709 AFFECTED FILES bin/hbase-config.sh bin/test/process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java
        Hide
        Phabricator added a comment -

        mbautin has commented on the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        Addressing Ted's comments. I will post a new version shortly.

        INLINE COMMENTS
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Even though ports for same-type servers in the process-based cluster are selected randomly, it would still make sense to call the lowest-port server #0, the next-lowest #1, etc.
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 Done.
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Done.
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Done.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        Show
        Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Addressing Ted's comments. I will post a new version shortly. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Even though ports for same-type servers in the process-based cluster are selected randomly, it would still make sense to call the lowest-port server #0, the next-lowest #1, etc. src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 Done. src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Done. src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Done. REVISION DETAIL https://reviews.facebook.net/D2709
        Hide
        Phabricator added a comment -

        tedyu has commented on the revision "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".

        Patch looks good.

        INLINE COMMENTS
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Should read 'to look up'
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Why is sorting needed ?
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 I think removing 'to kill' would make the sentence clearer.
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Should we add mapping for 'fs.defaultFS' as well ?

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        Show
        Phabricator added a comment - tedyu has commented on the revision " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Patch looks good. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java:44 Should read 'to look up' src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:114 Why is sorting needed ? src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:250 I think removing 'to kill' would make the sentence clearer. src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java:395 Should we add mapping for 'fs.defaultFS' as well ? REVISION DETAIL https://reviews.facebook.net/D2709
        Hide
        Phabricator added a comment -

        mbautin requested code review of "[jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust".
        Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA

        Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests.

        Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier.

        TEST PLAN
        Run the new unit test multiple times (10x or 50x).
        Run all unit tests.

        REVISION DETAIL
        https://reviews.facebook.net/D2709

        AFFECTED FILES
        bin/hbase-config.sh
        bin/test/kill_process_based_cluster.sh
        src/main/java/org/apache/hadoop/hbase/HConstants.java
        src/main/java/org/apache/hadoop/hbase/master/HMaster.java
        src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
        src/test/java/org/apache/hadoop/hbase/HTestConst.java
        src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
        src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
        src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java
        src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java
        src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
        src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
        src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java
        src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/6195/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - mbautin requested code review of " [jira] HBASE-5684 Make ProcessBasedLocalHBaseCluster run HDFS and make it more robust". Reviewers: Kannan, Liyin, khemani, Karthik, madhuvaidya, amirshim, JIRA Currently ProcessBasedLocalHBaseCluster runs on top of raw local filesystem. We need it to start a process-based HDFS cluster as well. We also need to make the whole thing more stable so we can use it in unit tests. Also all logs of local HBase cluster daemons are now tailed to the primary log of the unit test with the appropriate prefixes to make debugging easier. TEST PLAN Run the new unit test multiple times (10x or 50x). Run all unit tests. REVISION DETAIL https://reviews.facebook.net/D2709 AFFECTED FILES bin/hbase-config.sh bin/test/kill_process_based_cluster.sh src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/master/HMaster.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/HTestConst.java src/test/java/org/apache/hadoop/hbase/TestInfoServers.java src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestScanner.java src/test/java/org/apache/hadoop/hbase/util/HBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestHBaseHomePath.java src/test/java/org/apache/hadoop/hbase/util/TestProcessBasedCluster.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6195/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          People

          • Assignee:
            Mikhail Bautin
            Reporter:
            Mikhail Bautin
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development