Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None

      Description

      Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
      I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
      The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:

      hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP

      I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.

      1. ASF.LICENSE.NOT.GRANTED--D549.1.patch
        76 kB
        Phabricator
      2. ASF.LICENSE.NOT.GRANTED--D549.2.patch
        77 kB
        Phabricator
      3. ASF.LICENSE.NOT.GRANTED--D549.3.patch
        93 kB
        Phabricator
      4. ASF.LICENSE.NOT.GRANTED--D549.4.patch
        97 kB
        Phabricator
      5. ASF.LICENSE.NOT.GRANTED--D549.5.patch
        97 kB
        Phabricator
      6. ASF.LICENSE.NOT.GRANTED--D549.6.patch
        105 kB
        Phabricator
      7. ASF.LICENSE.NOT.GRANTED--D549.7.patch
        105 kB
        Phabricator
      8. 0001-HBase-cluster-test-tool.patch
        164 kB
        Mikhail Bautin
      9. ASF.LICENSE.NOT.GRANTED--D549.8.patch
        150 kB
        Phabricator
      10. 0002-HBase-cluster-test-tool.patch
        161 kB
        Mikhail Bautin
      11. ASF.LICENSE.NOT.GRANTED--D549.9.patch
        146 kB
        Phabricator
      12. 0003-HBase-cluster-test-tool.patch
        157 kB
        Mikhail Bautin

        Issue Links

          Activity

          Hide
          Ted Yu added a comment -

          We have OPT_ZK_QUORUM.
          I think adding a config parameter for zookeeper client port would make using this tool more easily.

          Show
          Ted Yu added a comment - We have OPT_ZK_QUORUM. I think adding a config parameter for zookeeper client port would make using this tool more easily.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #35 (See https://builds.apache.org/job/HBase-TRUNK-security/35/)
          HBASE-4935 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop – revert from TRUNK though this change was added as part of HBASE-4908

          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #35 (See https://builds.apache.org/job/HBase-TRUNK-security/35/ ) HBASE-4935 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop – revert from TRUNK though this change was added as part of HBASE-4908 stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2553 (See https://builds.apache.org/job/HBase-TRUNK/2553/)
          HBASE-4935 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop – revert from TRUNK though this change was added as part of HBASE-4908

          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2553 (See https://builds.apache.org/job/HBase-TRUNK/2553/ ) HBASE-4935 hbase 0.92.0 doesn't work going against 0.20.205.0, its packaged hadoop – revert from TRUNK though this change was added as part of HBASE-4908 stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          Hide
          stack added a comment -

          Backed out SequenceFileLogReader.java change – will address in a way that doesn't break us going against 0.22 and 0.23 in HBASE-4935

          Show
          stack added a comment - Backed out SequenceFileLogReader.java change – will address in a way that doesn't break us going against 0.22 and 0.23 in HBASE-4935
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2526 (See https://builds.apache.org/job/HBase-TRUNK/2526/)
          [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)

          Summary:
          Porting one of our HBase cluster test tools (a single-process multi-threaded
          load generator and verifier) from 0.89-fb to trunk.
          I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that
          it has some features that I have not tried yet (some kind of a kill test, and
          some way to run HBase as multiple processes on one machine).
          The main utility of this piece of code for us has been the HBaseClusterTest
          command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
          load test in our five-node dev cluster testing, e.g.:

          hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4
          -read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5
          -max_read_errors 10000 -num_keys 10000000000 -start_key 0

          Test Plan:
          Run this on a dev cluster. Run all unit tests.

          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Reviewed By: nspiegelberg

          CC: stack, nspiegelberg, mbautin, Karthik

          Differential Revision: 549

          nspiegelberg :
          Files :

          • /hbase/trunk/pom.xml
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Keying.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2526 (See https://builds.apache.org/job/HBase-TRUNK/2526/ ) [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb) Summary: Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk. I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine). The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.: hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4 -read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5 -max_read_errors 10000 -num_keys 10000000000 -start_key 0 Test Plan: Run this on a dev cluster. Run all unit tests. Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Reviewed By: nspiegelberg CC: stack, nspiegelberg, mbautin, Karthik Differential Revision: 549 nspiegelberg : Files : /hbase/trunk/pom.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HMerge.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Keying.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #25 (See https://builds.apache.org/job/HBase-TRUNK-security/25/)
          [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)

          Summary:
          Porting one of our HBase cluster test tools (a single-process multi-threaded
          load generator and verifier) from 0.89-fb to trunk.
          I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that
          it has some features that I have not tried yet (some kind of a kill test, and
          some way to run HBase as multiple processes on one machine).
          The main utility of this piece of code for us has been the HBaseClusterTest
          command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a
          load test in our five-node dev cluster testing, e.g.:

          hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4
          -read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5
          -max_read_errors 10000 -num_keys 10000000000 -start_key 0

          Test Plan:
          Run this on a dev cluster. Run all unit tests.

          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Reviewed By: nspiegelberg

          CC: stack, nspiegelberg, mbautin, Karthik

          Differential Revision: 549

          nspiegelberg :
          Files :

          • /hbase/trunk/pom.xml
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Keying.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #25 (See https://builds.apache.org/job/HBase-TRUNK-security/25/ ) [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb) Summary: Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk. I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine). The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.: hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4 -read 100:10 -zk <zk_quorum_node> -bloom ROWCOL -compression LZO -key_window 5 -max_read_errors 10000 -num_keys 10000000000 -start_key 0 Test Plan: Run this on a dev cluster. Run all unit tests. Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Reviewed By: nspiegelberg CC: stack, nspiegelberg, mbautin, Karthik Differential Revision: 549 nspiegelberg : Files : /hbase/trunk/pom.xml /hbase/trunk/src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HMerge.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Keying.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Phabricator added a comment -

          mbautin has committed the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          REVISION DETAIL
          https://reviews.facebook.net/D549

          COMMIT
          https://reviews.facebook.net/rHBASE1211746

          Show
          Phabricator added a comment - mbautin has committed the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". REVISION DETAIL https://reviews.facebook.net/D549 COMMIT https://reviews.facebook.net/rHBASE1211746
          Hide
          Phabricator added a comment -

          nspiegelberg has accepted the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          lgtm

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - nspiegelberg has accepted the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". lgtm REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          Some more unit tests results: 1189 test methods, 252 test classes succeeded. TestLogRolling failed when running on the cluster using mrunit, but succeeded locally.

          mvn -Dtest=TestLogRolling#testLogRollOnDatanodeDeath -P localTests test

          This patch should be OK to commit.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Some more unit tests results: 1189 test methods, 252 test classes succeeded. TestLogRolling failed when running on the cluster using mrunit, but succeeded locally. mvn -Dtest=TestLogRolling#testLogRollOnDatanodeDeath -P localTests test This patch should be OK to commit. REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506552/0003-HBase-cluster-test-tool.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 74 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -160 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 74 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/471//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/471//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/471//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506552/0003-HBase-cluster-test-tool.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 74 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 74 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/471//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/471//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/471//console This message is automatically generated.
          Hide
          Mikhail Bautin added a comment -

          One more round of Jenkins testing for the patch.

          Show
          Mikhail Bautin added a comment - One more round of Jenkins testing for the patch.
          Hide
          Mikhail Bautin added a comment -

          Attaching the most recent patch.

          Show
          Mikhail Bautin added a comment - Attaching the most recent patch.
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Fixing how reader and writer are linked together in the mixed workload. Verified that LoadTestTool works in both reader-only, writer-only, and writer/reader modes on a dev cluster. Unit test failures with and without the patch are identical. I think this patch is good to be committed.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Fixing how reader and writer are linked together in the mixed workload. Verified that LoadTestTool works in both reader-only, writer-only, and writer/reader modes on a dev cluster. Unit test failures with and without the patch are identical. I think this patch is good to be committed. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java src/main/java/org/apache/hadoop/hbase/util/Bytes.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestBytes.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506382/0002-HBase-cluster-test-tool.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 74 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -160 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 74 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/456//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/456//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/456//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506382/0002-HBase-cluster-test-tool.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 74 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 74 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/456//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/456//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/456//console This message is automatically generated.
          Hide
          Mikhail Bautin added a comment -

          Uploading a patch for Jenkins testing.

          Show
          Mikhail Bautin added a comment - Uploading a patch for Jenkins testing.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506381/D549.8.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 52 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/455//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506381/D549.8.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 52 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/455//console This message is automatically generated.
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Addressing Stack's and Nicolas's comments. Still need to re-run the unit tests.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Addressing Stack's and Nicolas's comments. Still need to re-run the unit tests. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java src/main/java/org/apache/hadoop/hbase/util/Bytes.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestBytes.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          Thanks for reviews, Nicolas and Stack! See responses below. A new version of the code will follow. Still have to re-run the unit tests (I've been having some trouble with those recently – http://pastebin.com/1G1ZcPeV) and sanity-check the command-line load tester. On a side note, here are some stats from my recent load test run on a 5-node cluster:

          11/12/06 18:30:25 INFO util.MultiThreadedAction: [W:21] Keys=17180542, cols=819.4m, time=27:49:20 Overall: [keys/s= 171, latency=116 ms] Current: [keys/s=219, latency=90 ms], insertedUpTo=17180495, insertedQSize=26
          11/12/06 18:30:25 INFO util.MultiThreadedAction: [R:10] Keys=250690067, cols=11.8g, time=27:49:20 Overall: [keys/s= 2502, latency=3 ms] Current: [keys/s=261, latency=38 ms], verified=250690067

          (The number of writer's threads is reported as 21 because there is an "inserted keys tracker" thread that keeps track of the most recent contiguous key written by all writers.)

          INLINE COMMENTS
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 Changed this to 80. There is no standard way to get terminal width in Java. http://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java

          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 This is the command-line part of LoadTest, so it should not run as part of the test suite. The same multithreaded writer/verifier code is reused in a couple of "large" unit tests, namely TestMiniClusterLoad

          {Parallel,Sequential}

          .

          I added a line about the differences between this load tester and PerformanceEvaluation.

          Renamed this to LoadTestTool.
          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 We first wait for all writers to finish and then wait for all readers to finish, so we should exit when all of those threads stop.
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Yes, this can be converted to an IntegrationTest. I think making this a unit test was proposed in the past but the controversy was that it spawns a bunch of child processes and extra care should be taken to shut down all of them. We can leave this as a command-line tool for now and convert to a unit test when it is more stable.
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 Removed these methods and reused the existing RegionSetting.HexStringSplit class.
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 I moved this to from the test jar to the main jar because of the waitForBaseZNode method that I added to ZKUtil. We need this kind of watcher for quick ZK checks when we don't want to use a callback. Passing a null instead of a watcher does not work.
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Not sure what changes you are talking about. I made a few conf options that I needed to use constants in HConstants.
          src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 Removed this.
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 Done. Thanks!
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Renamed this to AbstractHBaseTool and moved to the main jar, since this is general-purpose command line processing functionality. This also avoids the confusion with the IntegrationTest notation (thanks for the link!) I think we will be creating a lot of integration tests very soon.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Thanks for reviews, Nicolas and Stack! See responses below. A new version of the code will follow. Still have to re-run the unit tests (I've been having some trouble with those recently – http://pastebin.com/1G1ZcPeV ) and sanity-check the command-line load tester. On a side note, here are some stats from my recent load test run on a 5-node cluster: 11/12/06 18:30:25 INFO util.MultiThreadedAction: [W:21] Keys=17180542, cols=819.4m, time=27:49:20 Overall: [keys/s= 171, latency=116 ms] Current: [keys/s=219, latency=90 ms] , insertedUpTo=17180495, insertedQSize=26 11/12/06 18:30:25 INFO util.MultiThreadedAction: [R:10] Keys=250690067, cols=11.8g, time=27:49:20 Overall: [keys/s= 2502, latency=3 ms] Current: [keys/s=261, latency=38 ms] , verified=250690067 (The number of writer's threads is reported as 21 because there is an "inserted keys tracker" thread that keeps track of the most recent contiguous key written by all writers.) INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 Changed this to 80. There is no standard way to get terminal width in Java. http://stackoverflow.com/questions/1286461/can-i-find-the-console-width-with-java src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 This is the command-line part of LoadTest, so it should not run as part of the test suite. The same multithreaded writer/verifier code is reused in a couple of "large" unit tests, namely TestMiniClusterLoad {Parallel,Sequential} . I added a line about the differences between this load tester and PerformanceEvaluation. Renamed this to LoadTestTool. src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 We first wait for all writers to finish and then wait for all readers to finish, so we should exit when all of those threads stop. src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Yes, this can be converted to an IntegrationTest. I think making this a unit test was proposed in the past but the controversy was that it spawns a bunch of child processes and extra care should be taken to shut down all of them. We can leave this as a command-line tool for now and convert to a unit test when it is more stable. src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 Removed these methods and reused the existing RegionSetting.HexStringSplit class. src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 I moved this to from the test jar to the main jar because of the waitForBaseZNode method that I added to ZKUtil. We need this kind of watcher for quick ZK checks when we don't want to use a callback. Passing a null instead of a watcher does not work. src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Not sure what changes you are talking about. I made a few conf options that I needed to use constants in HConstants. src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 Removed this. src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 Done. Thanks! src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Renamed this to AbstractHBaseTool and moved to the main jar, since this is general-purpose command line processing functionality. This also avoids the confusion with the IntegrationTest notation (thanks for the link!) I think we will be creating a lot of integration tests very soon. REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          This is looking great Mikhail.

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 When would I want one of these?
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Is this from another issue Mikhail? (No matter if it is... we can take care of it on commit).
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java:37 Thanks for doing this.
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java:111 Thanks for fixing this.
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1798 Good
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Will this be run as an IntegrationTest because it has the IntegrationTest prefix; (see the Integration Test section on this page, http://hbase.apache.org/book/hbase.tests.html, if you don't have cluse what I'm on about)?

          I like the idea of this class. We need it. Nice how you subclass tool.
          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 Should this class have the IntegrationTest prefix or you think this is just a tool not part of IntegrationTests?

          Since its sitting beside PerformanceEvaluation tool, should you say something on how it differs from it?
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Should this be an IntegrationTest? (We can do the convertion in another issue).

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". This is looking great Mikhail. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java:28 When would I want one of these? src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java:176 Is this from another issue Mikhail? (No matter if it is... we can take care of it on commit). src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java:37 Thanks for doing this. src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java:111 Thanks for fixing this. src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1798 Good src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:39 Will this be run as an IntegrationTest because it has the IntegrationTest prefix; (see the Integration Test section on this page, http://hbase.apache.org/book/hbase.tests.html , if you don't have cluse what I'm on about)? I like the idea of this class. We need it. Nice how you subclass tool. src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:40 Should this class have the IntegrationTest prefix or you think this is just a tool not part of IntegrationTests? Since its sitting beside PerformanceEvaluation tool, should you say something on how it differs from it? src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java:37 Should this be an IntegrationTest? (We can do the convertion in another issue). REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          Karthik has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          INLINE COMMENTS
          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 One question - when does this integration test exit? Both for the write and read paths?

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - Karthik has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/LoadTest.java:318 One question - when does this integration test exit? Both for the write and read paths? REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12506189/0001-HBase-cluster-test-tool.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 77 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated -160 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 72 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.util.TestRegionSplitter
          org.apache.hadoop.hbase.client.TestMultiParallel
          org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler
          org.apache.hadoop.hbase.regionserver.wal.TestHLogBench
          org.apache.hadoop.hbase.rest.TestGzipFilter
          org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
          org.apache.hadoop.hbase.regionserver.TestAtomicOperation
          org.apache.hadoop.hbase.rest.TestScannersWithFilters
          org.apache.hadoop.hbase.TestInfoServers
          org.apache.hadoop.hbase.regionserver.TestParallelPut
          org.apache.hadoop.hbase.coprocessor.TestClassLoading
          org.apache.hadoop.hbase.filter.TestColumnRangeFilter
          org.apache.hadoop.hbase.client.TestHCM
          org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
          org.apache.hadoop.hbase.coprocessor.TestMasterObserver
          org.apache.hadoop.hbase.rest.TestStatusResource
          org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort
          org.apache.hadoop.hbase.rest.TestVersionResource
          org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
          org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
          org.apache.hadoop.hbase.rest.TestRowResource
          org.apache.hadoop.hbase.rest.TestScannerResource
          org.apache.hadoop.hbase.ipc.TestDelayedRpc
          org.apache.hadoop.hbase.rest.client.TestRemoteAdmin
          org.apache.hadoop.hbase.util.TestFSUtils
          org.apache.hadoop.hbase.rest.TestTableResource
          org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
          org.apache.hadoop.hbase.util.TestIdLock
          org.apache.hadoop.hbase.rest.TestTransform
          org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
          org.apache.hadoop.hbase.regionserver.TestHRegion
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
          org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
          org.apache.hadoop.hbase.client.TestMetaScanner
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock
          org.apache.hadoop.hbase.client.TestTimestampsFilter
          org.apache.hadoop.hbase.master.TestMaster
          org.apache.hadoop.hbase.TestDrainingServer
          org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
          org.apache.hadoop.hbase.rest.TestSchemaResource
          org.apache.hadoop.hbase.TestAcidGuarantees
          org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
          org.apache.hadoop.hbase.avro.TestAvroServer
          org.apache.hadoop.hbase.rest.client.TestRemoteTable
          org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
          org.apache.hadoop.hbase.util.TestHBaseFsck
          org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove
          org.apache.hadoop.hbase.client.TestHTableUtil
          org.apache.hadoop.hbase.thrift.TestThriftServer
          org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
          org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics
          org.apache.hadoop.hbase.util.TestMergeTable
          org.apache.hadoop.hbase.mapreduce.TestImportTsv
          org.apache.hadoop.hbase.rest.TestMultiRowResource
          org.apache.hadoop.hbase.TestMultiVersions
          org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
          org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove
          org.apache.hadoop.hbase.coprocessor.TestWALObserver
          org.apache.hadoop.hbase.master.TestOpenedRegionHandler
          org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/450//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/450//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/450//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12506189/0001-HBase-cluster-test-tool.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 77 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -160 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 72 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.util.TestRegionSplitter org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler org.apache.hadoop.hbase.regionserver.wal.TestHLogBench org.apache.hadoop.hbase.rest.TestGzipFilter org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.rest.TestScannersWithFilters org.apache.hadoop.hbase.TestInfoServers org.apache.hadoop.hbase.regionserver.TestParallelPut org.apache.hadoop.hbase.coprocessor.TestClassLoading org.apache.hadoop.hbase.filter.TestColumnRangeFilter org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.rest.TestStatusResource org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort org.apache.hadoop.hbase.rest.TestVersionResource org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol org.apache.hadoop.hbase.rest.TestRowResource org.apache.hadoop.hbase.rest.TestScannerResource org.apache.hadoop.hbase.ipc.TestDelayedRpc org.apache.hadoop.hbase.rest.client.TestRemoteAdmin org.apache.hadoop.hbase.util.TestFSUtils org.apache.hadoop.hbase.rest.TestTableResource org.apache.hadoop.hbase.regionserver.wal.TestWALReplay org.apache.hadoop.hbase.util.TestIdLock org.apache.hadoop.hbase.rest.TestTransform org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.catalog.TestMetaReaderEditor org.apache.hadoop.hbase.client.TestMetaScanner org.apache.hadoop.hbase.io.hfile.TestHFileBlock org.apache.hadoop.hbase.client.TestTimestampsFilter org.apache.hadoop.hbase.master.TestMaster org.apache.hadoop.hbase.TestDrainingServer org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass org.apache.hadoop.hbase.rest.TestSchemaResource org.apache.hadoop.hbase.TestAcidGuarantees org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase org.apache.hadoop.hbase.avro.TestAvroServer org.apache.hadoop.hbase.rest.client.TestRemoteTable org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol org.apache.hadoop.hbase.util.TestHBaseFsck org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove org.apache.hadoop.hbase.client.TestHTableUtil org.apache.hadoop.hbase.thrift.TestThriftServer org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics org.apache.hadoop.hbase.util.TestMergeTable org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.rest.TestMultiRowResource org.apache.hadoop.hbase.TestMultiVersions org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove org.apache.hadoop.hbase.coprocessor.TestWALObserver org.apache.hadoop.hbase.master.TestOpenedRegionHandler org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/450//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/450//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/450//console This message is automatically generated.
          Hide
          Phabricator added a comment -

          nspiegelberg has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          INLINE COMMENTS
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 isn't the standard width 80? Is there a system setting we can use?

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - nspiegelberg has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java:129 isn't the standard width 80? Is there a system setting we can use? REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          nspiegelberg has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          INLINE COMMENTS
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 this seems util-specific. this should probably be in some more specialized class rather than Bytes.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 all the changes in Keying.java are already found in RegionSplitter.java under HexStringSplit. I would reuse that class
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 better to use

          splits = RegionSplitter.HexStringSplit.split(totalNumberOfRegions)

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - nspiegelberg has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/util/Bytes.java:1658-1680 this seems util-specific. this should probably be in some more specialized class rather than Bytes.java src/main/java/org/apache/hadoop/hbase/util/Keying.java:41-43 all the changes in Keying.java are already found in RegionSplitter.java under HexStringSplit. I would reuse that class src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:1820-1821 better to use splits = RegionSplitter.HexStringSplit.split(totalNumberOfRegions) REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Rebased on changes from trunk.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Rebased on changes from trunk. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/Bytes.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java src/test/java/org/apache/hadoop/hbase/util/LoadTest.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestBytes.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          The unit tests run fine (except TestReplication.queueFailover, which is flaky), and cluster testing works well. I rewrote the writer/reader synchronization mechanism after talking to Kannan, so that the reader gets the precise knowledge of what is the last key in the contiguous sequence of keys starting from startKey that the writer has written. This required implementing a producer/consumer pattern with a special InsertedKeyTracker thread sorting inserted keys and advancing the "last contiguous key" marker. Also clarified and improved a bunch of command-line stuff in LoadTest.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA The unit tests run fine (except TestReplication.queueFailover, which is flaky), and cluster testing works well. I rewrote the writer/reader synchronization mechanism after talking to Kannan, so that the reader gets the precise knowledge of what is the last key in the contiguous sequence of keys starting from startKey that the writer has written. This required implementing a producer/consumer pattern with a special InsertedKeyTracker thread sorting inserted keys and advancing the "last contiguous key" marker. Also clarified and improved a bunch of command-line stuff in LoadTest. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/Bytes.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java src/test/java/org/apache/hadoop/hbase/util/LoadTest.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestBytes.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          This passes unit tests (I will confirm this again – please do not commit yet), and is successfully running a load test in a 5-node cluster.

          The reader verifies keys sequentially in the current version, and if it catches up with the reader it sleeps until more keys are written. This is different from the original version of HBaseTest in 0.89-fb (renamed LoadTest in this version), and I think I will modify the reader to read some previous keys that are known to have been written when the "current key" of the reader catches up with the last key written by the writer.

          I also made the number of read errors to tolerate configurable (the --max_read_errors option).

          I think it will take me a couple more iterations on this diff before it can be committed. Sorry for a lot of updates and emails.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/Bytes.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTest.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
          src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA This passes unit tests (I will confirm this again – please do not commit yet), and is successfully running a load test in a 5-node cluster. The reader verifies keys sequentially in the current version, and if it catches up with the reader it sleeps until more keys are written. This is different from the original version of HBaseTest in 0.89-fb (renamed LoadTest in this version), and I think I will modify the reader to read some previous keys that are known to have been written when the "current key" of the reader catches up with the last key written by the writer. I also made the number of read errors to tolerate configurable (the --max_read_errors option). I think it will take me a couple more iterations on this diff before it can be committed. Sorry for a lot of updates and emails. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/Bytes.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java src/test/java/org/apache/hadoop/hbase/util/LoadTest.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestBytes.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          @stack: see responses to your comments inline. A new version of the diff will follow.

          INLINE COMMENTS
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:47 Added class documentation.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:52 Removed trailing underscores.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:275 I added some class comment anyway, and removed kill test and append test from this tool (we can port them over separately).
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:38 Moved these tool methods to Keying and HBaseTestingUtility and got rid of this class.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:53 Got rid of this method entirely and started using the HTable constructor directly.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:54 See above (got rid of this method).
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:59 Moved this to HBaseTestingUtility.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:92 Changed to "getMetaRSPort" (because we only need the port number in this use case) and modified to use getRegionLocation.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:96 See the above comment.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:99 Got rid of this method.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:111 Moved to Keying. Not sure where HBaseSplitter is (could not find that type or file).
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:124 Moved to Keying.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:139 Moved to Bytes.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:151 Moved to Bytes.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java:36 Removed HdfsAppender from this diff.
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:33 Removed the kill test from this diff.
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:119 Removed this entire class from the diff for now.
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:152 Removed this entire class (kill test) for now.
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java:31 Added.
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedActionMBean.java:19 Removed this class. Its only purpose was to collect stats from the distributed load test through JMX.
          src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:36 Added a class comment. Unlike MiniHBaseCluster, this starts each daemon as a separate process and allows to do real kill testing.
          src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:256 Refactored this to generate configuration from a map. Does that address your concern, or are you objecting to the whole idea of auto-generating configuration for daemon processes in a process-based local HBase cluster?

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". @stack: see responses to your comments inline. A new version of the diff will follow. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:47 Added class documentation. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:52 Removed trailing underscores. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:275 I added some class comment anyway, and removed kill test and append test from this tool (we can port them over separately). src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:38 Moved these tool methods to Keying and HBaseTestingUtility and got rid of this class. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:53 Got rid of this method entirely and started using the HTable constructor directly. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:54 See above (got rid of this method). src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:59 Moved this to HBaseTestingUtility. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:92 Changed to "getMetaRSPort" (because we only need the port number in this use case) and modified to use getRegionLocation. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:96 See the above comment. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:99 Got rid of this method. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:111 Moved to Keying. Not sure where HBaseSplitter is (could not find that type or file). src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:124 Moved to Keying. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:139 Moved to Bytes. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:151 Moved to Bytes. src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java:36 Removed HdfsAppender from this diff. src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:33 Removed the kill test from this diff. src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:119 Removed this entire class from the diff for now. src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:152 Removed this entire class (kill test) for now. src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java:31 Added. src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedActionMBean.java:19 Removed this class. Its only purpose was to collect stats from the distributed load test through JMX. src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:36 Added a class comment. Unlike MiniHBaseCluster, this starts each daemon as a separate process and allows to do real kill testing. src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:256 Refactored this to generate configuration from a map. Does that address your concern, or are you objecting to the whole idea of auto-generating configuration for daemon processes in a process-based local HBase cluster? REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          mbautin has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          Some replies inline. I will remove HdfsAppender test as well – it seems unrelated to the main purpose of the load tester tool.

          INLINE COMMENTS
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 Moved to org.apache.hadoop.hbase.util.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 Done.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 Made 20 a constant and removed separate logic for the (endKey - startKey) <= 1000 case.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Mentioned this in the class comment.
          src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Added.
          src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is not a lot of overlap of functionality, but perhaps some overlap in structure, e.g. command-line argument processing. I'd like to avoid doing a lot of refactoring until the next iteration. Ideally it should be possible to run every integration test both as a JUnit test and as a command-line tool.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - mbautin has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Some replies inline. I will remove HdfsAppender test as well – it seems unrelated to the main purpose of the load tester tool. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 Moved to org.apache.hadoop.hbase.util. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 Done. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 Made 20 a constant and removed separate logic for the (endKey - startKey) <= 1000 case. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Mentioned this in the class comment. src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Added. src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is not a lot of overlap of functionality, but perhaps some overlap in structure, e.g. command-line argument processing. I'd like to avoid doing a lot of refactoring until the next iteration. Ideally it should be possible to run every integration test both as a JUnit test and as a command-line tool. REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Addressed some more of Stack's comments. Got rid of the kill test – it is a large area by itself and is best left out of the scope of this diff. Refactored the common part of RestartMetaTest and HBaseClusterTest into IntegrationTestTool. I think eventually we need to be able to run integration tests both from command line and through the JUnit framework. Also, we can probably unify the mini-cluster framework with the process-based HBase cluster stuff later, and have a subset of integration tests run in both modes.

          I also added two unit tests: TestMiniClusterLoad

          {Sequential,Parallel}

          . The former runs a write workload followed by a separate read workload, and the latter one runs a read workload concurrently with the writes (which is noticeably slower – maybe that's something to investigate).

          A thorough round of testing and cleanup is still necessary before this diff can be finalized.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/Keying.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
          src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
          src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
          src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java
          src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Addressed some more of Stack's comments. Got rid of the kill test – it is a large area by itself and is best left out of the scope of this diff. Refactored the common part of RestartMetaTest and HBaseClusterTest into IntegrationTestTool. I think eventually we need to be able to run integration tests both from command line and through the JUnit framework. Also, we can probably unify the mini-cluster framework with the process-based HBase cluster stuff later, and have a subset of integration tests run in both modes. I also added two unit tests: TestMiniClusterLoad {Sequential,Parallel} . The former runs a write workload followed by a separate read workload, and the latter one runs a read workload concurrently with the writes (which is noticeably slower – maybe that's something to investigate). A thorough round of testing and cleanup is still necessary before this diff can be finalized. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/Keying.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java src/test/java/org/apache/hadoop/hbase/util/IntegrationTestTool.java src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Debugged and significantly improved RestartMetaTest and its dependency ProcessBasedLocalHBaseCluster to the point of actually being able to run a local multi-process test of an HBase cluster that kills the region server with the .META. region and splits the logs. Discovered that our log splitting does not work correctly with RawLocalFileSystem or checksum-enabled local filesystem, and added a fix to the hack in SequenceFileLogReader to only apply reflection logic to DFSInputStream.

          Still need to clean up the code, re-run unit tests, and re-test HBaseClusterTest on a live cluster.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          pom.xml
          src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
          src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
          src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
          src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
          src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
          src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Debugged and significantly improved RestartMetaTest and its dependency ProcessBasedLocalHBaseCluster to the point of actually being able to run a local multi-process test of an HBase cluster that kills the region server with the .META. region and splits the logs. Discovered that our log splitting does not work correctly with RawLocalFileSystem or checksum-enabled local filesystem, and added a fix to the hack in SequenceFileLogReader to only apply reflection logic to DFSInputStream. Still need to clean up the code, re-run unit tests, and re-test HBaseClusterTest on a live cluster. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES pom.xml src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java src/test/java/org/apache/hadoop/hbase/EmptyWatcher.java src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          Hide
          Phabricator added a comment -

          mbautin updated the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Partially addressing Stack's comments (I will address the rest of them in the next version of this diff). The load test seems to work fine on a dev cluster. I also tried out RestartMetaTest. The idea is to start a local HBase cluster with multiple regionservers on different ports, load some data, kill a regionserver, restart it, and do some checks. I like the of a "process-based local HBase cluster" because it opens up a possibility for a whole new range of unit tests (integration tests?) where we are allowed to kill processes for real, but cannot inspect the internals of those processes. This also allows us to test cluster management scripts (the hbase script, hbase-daemon.sh, etc.) from Java itself. Of course, we need to make sure we don't leave those processes hanging, but that can be addressed in the shutdown hook of the unit test (or @After handler).

          RestartMetaTest still needs some work to make it robust and usable, and we probably want to make it a unit test.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          src/main/java/org/apache/hadoop/hbase/HConstants.java
          src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
          src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
          src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
          src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
          src/main/java/org/apache/hadoop/hbase/util/HMerge.java
          src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
          src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
          src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
          src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
          src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
          src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java
          src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java
          src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java
          src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java
          src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java
          src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java

          Show
          Phabricator added a comment - mbautin updated the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Partially addressing Stack's comments (I will address the rest of them in the next version of this diff). The load test seems to work fine on a dev cluster. I also tried out RestartMetaTest. The idea is to start a local HBase cluster with multiple regionservers on different ports, load some data, kill a regionserver, restart it, and do some checks. I like the of a "process-based local HBase cluster" because it opens up a possibility for a whole new range of unit tests (integration tests?) where we are allowed to kill processes for real, but cannot inspect the internals of those processes. This also allows us to test cluster management scripts (the hbase script, hbase-daemon.sh, etc.) from Java itself. Of course, we need to make sure we don't leave those processes hanging, but that can be addressed in the shutdown hook of the unit test (or @After handler). RestartMetaTest still needs some work to make it robust and usable, and we probably want to make it a unit test. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/HConstants.java src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java src/main/java/org/apache/hadoop/hbase/util/HMerge.java src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java src/test/java/org/apache/hadoop/hbase/util/ClusterTestUtils.java src/test/java/org/apache/hadoop/hbase/util/DataGenerator.java src/test/java/org/apache/hadoop/hbase/util/HBaseClusterTest.java src/test/java/org/apache/hadoop/hbase/util/HdfsAppender.java src/test/java/org/apache/hadoop/hbase/util/KillProcessesAndVerify.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/util/ProcessBasedLocalHBaseCluster.java src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          @Nicolas Sure on the distinction but PE can do threaded or MR'd workloads and has a means of plugging in different loading profiles (subclass Client and then implement run). I'd think it would be better able to put up a sustained loading when loading is coming out of MR (in spite of its ugly name – name comes from BT paper).

          I'm fine w/ this tool going in as is (after some cleanup). Can work then separately on unifying the two if it makes sense as a different project.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". @Nicolas Sure on the distinction but PE can do threaded or MR'd workloads and has a means of plugging in different loading profiles (subclass Client and then implement run). I'd think it would be better able to put up a sustained loading when loading is coming out of MR (in spite of its ugly name – name comes from BT paper). I'm fine w/ this tool going in as is (after some cleanup). Can work then separately on unifying the two if it makes sense as a different project. REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          nspiegelberg has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          @stack : We should have a distinction:

          PerformanceEvaluation - meant to get an idea of the performance of your system setup. Able to find performance regressions or understand theoretical performance gains of datapath changes

          Load Tester - meant to provide deterministic, long-running load on a system. Allows for large scale verification to catch race conditions resulting from logic error or process failure (in a distributed system setting).

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - nspiegelberg has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". @stack : We should have a distinction: PerformanceEvaluation - meant to get an idea of the performance of your system setup. Able to find performance regressions or understand theoretical performance gains of datapath changes Load Tester - meant to provide deterministic, long-running load on a system. Allows for large scale verification to catch race conditions resulting from logic error or process failure (in a distributed system setting). REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          stack has commented on the revision "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".

          Kinda close to PE but I suppose different enough. This can give constant update on rates of read and write. Could add that to PE. Also, can do load+read at same time. Would have to do that in different threads in PE. Has more variable load too.

          INLINE COMMENTS
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 is this the right package M? util?
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:47 A bit of class doc on what it is might help.

          Might also just point them at usage it seems since that looks like it might be pretty good.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:52 No biggie but these data member names with a trailing underscore a bit silly (smile)
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:58 I suppose its fine this is hard-coded for now.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 No biggie: check cols is at least 3 in size so we can return 'nicer' message if less than 3
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 This hard coding should be explained somewhere? Maybe in class comment? The 20/1. Or in usage.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Yeah, I think these 'features' need to bubble it up to the class comment so folks can see whats in here w/o having to read code.
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:275 Hmm... maybe this is good enough when it prints out for figuring whats in here?
          src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Class comment
          src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:17 ditto on package. Should this be in under src/test or under src/main? Maybe package should be tool rather than manual?
          src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is a lot of overlap w/ previous test here, the HBaseClusterTest tool?
          src/test/java/org/apache/hadoop/hbase/manual/utils/DataGenerator.java:24 comment on what this does?
          src/test/java/org/apache/hadoop/hbase/manual/utils/DisplayFormatUtils.java:24 This class not needed? Use http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/StringUtils.html#humanReadableInt(long)
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:38 Class comment. HBaseUtils is a pretty generic name; says nothing about content.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:53 Remove
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:54 Why not just let this out? Should at least say in javadoc that it can return a null?
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:59 This method does not look like it belongs in a generic hbaseutils class. It looks like it belongs to the testing tool that makes up the bulk of this patch. Doesn't seem like method name describes properly what it does either. Why not let out exceptions rather than catch and log.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:92 HSA is deprecated.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:96 Use MetaReader? e.g:

          public static Pair<HRegionInfo, ServerName> getRegion(
          CatalogTracker catalogTracker, byte [] regionName)
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:99 Odd name for this method and it also looks a bit dodgy returning an HBC but it only has zk location set into it.
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:111 This looks like it belongs in Keying class under util? Or in HBaseSplitter?
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:124 ditto
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:139 This should be in Bytes (is it not already?)
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:151 ditto
          src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java:36 Does this belong in an hbase test class?
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:33 class comment.
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:119 Let these out rather than just print them?
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:152 What keys are being passed here?
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java:31 Class comment
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedReader.java:156 Remove
          src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:36 Class comment. Is this doing what minihbasecluster does?
          src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:256 Not sure this a good idea.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          Show
          Phabricator added a comment - stack has commented on the revision " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Kinda close to PE but I suppose different enough. This can give constant update on rates of read and write. Could add that to PE. Also, can do load+read at same time. Would have to do that in different threads in PE. Has more variable load too. INLINE COMMENTS src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:17 is this the right package M? util? src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:47 A bit of class doc on what it is might help. Might also just point them at usage it seems since that looks like it might be pretty good. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:52 No biggie but these data member names with a trailing underscore a bit silly (smile) src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:58 I suppose its fine this is hard-coded for now. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:92 No biggie: check cols is at least 3 in size so we can return 'nicer' message if less than 3 src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:103 This hard coding should be explained somewhere? Maybe in class comment? The 20/1. Or in usage. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:204 Yeah, I think these 'features' need to bubble it up to the class comment so folks can see whats in here w/o having to read code. src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java:275 Hmm... maybe this is good enough when it prints out for figuring whats in here? src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:36 Class comment src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:17 ditto on package. Should this be in under src/test or under src/main? Maybe package should be tool rather than manual? src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java:152 There is a lot of overlap w/ previous test here, the HBaseClusterTest tool? src/test/java/org/apache/hadoop/hbase/manual/utils/DataGenerator.java:24 comment on what this does? src/test/java/org/apache/hadoop/hbase/manual/utils/DisplayFormatUtils.java:24 This class not needed? Use http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/util/StringUtils.html#humanReadableInt(long ) src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:38 Class comment. HBaseUtils is a pretty generic name; says nothing about content. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:53 Remove src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:54 Why not just let this out? Should at least say in javadoc that it can return a null? src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:59 This method does not look like it belongs in a generic hbaseutils class. It looks like it belongs to the testing tool that makes up the bulk of this patch. Doesn't seem like method name describes properly what it does either. Why not let out exceptions rather than catch and log. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:92 HSA is deprecated. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:96 Use MetaReader? e.g: public static Pair<HRegionInfo, ServerName> getRegion( CatalogTracker catalogTracker, byte [] regionName) src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:99 Odd name for this method and it also looks a bit dodgy returning an HBC but it only has zk location set into it. src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:111 This looks like it belongs in Keying class under util? Or in HBaseSplitter? src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:124 ditto src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:139 This should be in Bytes (is it not already?) src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java:151 ditto src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java:36 Does this belong in an hbase test class? src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:33 class comment. src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:119 Let these out rather than just print them? src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java:152 What keys are being passed here? src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java:31 Class comment src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedReader.java:156 Remove src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:36 Class comment. Is this doing what minihbasecluster does? src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java:256 Not sure this a good idea. REVISION DETAIL https://reviews.facebook.net/D549
          Hide
          Phabricator added a comment -

          mbautin requested code review of "[jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)".
          Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA

          Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk.
          I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine).
          The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.:

          hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP

          I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works.

          TEST PLAN
          Run this on a dev cluster. Explore various modes of the tool (I've only tried the load test mode before). Probably make a unit test that runs a small local load test.

          REVISION DETAIL
          https://reviews.facebook.net/D549

          AFFECTED FILES
          src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java
          src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/DataGenerator.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/DisplayFormatUtils.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedActionMBean.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedReader.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedWriter.java
          src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/1227/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - mbautin requested code review of " [jira] HBASE-4908 HBase cluster test tool (port from 0.89-fb)". Reviewers: stack, Karthik, Kannan, nspiegelberg, JIRA Porting one of our HBase cluster test tools (a single-process multi-threaded load generator and verifier) from 0.89-fb to trunk. I cleaned up the code a bit compared to what's in 0.89-fb, and discovered that it has some features that I have not tried yet (some kind of a kill test, and some way to run HBase as multiple processes on one machine). The main utility of this piece of code for us has been the HBaseClusterTest command-line tool (called HBaseTest in 0.89-fb), which we usually invoke as a load test in our five-node dev cluster testing, e.g.: hbase org.apache.hadoop.hbase.manual.HBaseTest -load 1000000000:50:100:20 -tn load_test -read 1:1000000000:50:20 -zk <zk_quorum> -bloom ROWCOL -compression GZIP I will be using this code to load-test the delta encoding patch and making fixes, but I am submitting the patch for early feedback. I will probably try out its other functionality and comment on how it works. TEST PLAN Run this on a dev cluster. Explore various modes of the tool (I've only tried the load test mode before). Probably make a unit test that runs a small local load test. REVISION DETAIL https://reviews.facebook.net/D549 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/manual/HBaseClusterTest.java src/test/java/org/apache/hadoop/hbase/manual/RestartMetaTest.java src/test/java/org/apache/hadoop/hbase/manual/utils/DataGenerator.java src/test/java/org/apache/hadoop/hbase/manual/utils/DisplayFormatUtils.java src/test/java/org/apache/hadoop/hbase/manual/utils/HBaseUtils.java src/test/java/org/apache/hadoop/hbase/manual/utils/HdfsAppender.java src/test/java/org/apache/hadoop/hbase/manual/utils/KillProcessesAndVerify.java src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedAction.java src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedActionMBean.java src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedReader.java src/test/java/org/apache/hadoop/hbase/manual/utils/MultiThreadedWriter.java src/test/java/org/apache/hadoop/hbase/manual/utils/ProcessBasedLocalHBaseCluster.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1227/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.

            People

            • Assignee:
              Mikhail Bautin
              Reporter:
              Mikhail Bautin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development