Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.92.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      LoadTestTool is a single-process multi-threaded load generator and verifier designed to be invoked as a load test in a dev cluster, e.g.: hbase org.apache.hadoop.hbase.util.LoadTestTool -start_key 0 -num_keys 1000000000000 -bloom ROWCOL -compression NONE -write 10:100:100 -read 100:20 -tn loadtest1. Invoke without options for usage.
      Show
      LoadTestTool is a single-process multi-threaded load generator and verifier designed to be invoked as a load test in a dev cluster, e.g.: hbase org.apache.hadoop.hbase.util.LoadTestTool -start_key 0 -num_keys 1000000000000 -bloom ROWCOL -compression NONE -write 10:100:100 -read 100:20 -tn loadtest1. Invoke without options for usage.

      Description

      LoadTestTool is very useful.
      This JIRA backports LoadTestTool to 0.92 so that users don't have to build TRUNK in order to use it against 0.92 cluster.

      1. 5124-v4.patch
        67 kB
        Andrew Purtell
      2. 5124-v5.patch
        65 kB
        Andrew Purtell
      3. hbase-5124.txt
        54 kB
        Ted Yu
      4. hbase-5124-v3.txt
        54 kB
        Ted Yu

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.92-security #107 (See https://builds.apache.org/job/HBase-0.92-security/107/)
          HBASE-5124. Backport LoadTestTool to 0.92 (Revision 1339890)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Show
          Hudson added a comment - Integrated in HBase-0.92-security #107 (See https://builds.apache.org/job/HBase-0.92-security/107/ ) HBASE-5124 . Backport LoadTestTool to 0.92 (Revision 1339890) Result = FAILURE apurtell : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.92 #410 (See https://builds.apache.org/job/HBase-0.92/410/)
          HBASE-5124. Backport LoadTestTool to 0.92 (Revision 1339890)

          Result = FAILURE
          apurtell :
          Files :

          • /hbase/branches/0.92/CHANGES.txt
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java
          • /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java
          • /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Show
          Hudson added a comment - Integrated in HBase-0.92 #410 (See https://builds.apache.org/job/HBase-0.92/410/ ) HBASE-5124 . Backport LoadTestTool to 0.92 (Revision 1339890) Result = FAILURE apurtell : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Threads.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadParallel.java /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java
          Hide
          Andrew Purtell added a comment -

          Committed v5. TestMiniClusterLoadParallel passed 100 times locally.

          Show
          Andrew Purtell added a comment - Committed v5. TestMiniClusterLoadParallel passed 100 times locally.
          Hide
          Ted Yu added a comment -

          Patch v5 looks good (I didn't run test suite).

          Show
          Ted Yu added a comment - Patch v5 looks good (I didn't run test suite).
          Hide
          Andrew Purtell added a comment -

          v5 patch passes all new tests locally, removes superfluous unit test (no block encoding options in 0.92).

          Please ack for commit.

          Show
          Andrew Purtell added a comment - v5 patch passes all new tests locally, removes superfluous unit test (no block encoding options in 0.92). Please ack for commit.
          Hide
          Andrew Purtell added a comment -

          Getting test failures, looking into it.

          Show
          Andrew Purtell added a comment - Getting test failures, looking into it.
          Hide
          Andrew Purtell added a comment -

          v4 patch also ports over tests, does not include HBASE-6021 fixes, I'll commit those over this after commit. Running tests now.

          Show
          Andrew Purtell added a comment - v4 patch also ports over tests, does not include HBASE-6021 fixes, I'll commit those over this after commit. Running tests now.
          Hide
          Andrew Purtell added a comment -

          Also bringing over tests missing from v3 patch.

          Show
          Andrew Purtell added a comment - Also bringing over tests missing from v3 patch.
          Hide
          Andrew Purtell added a comment - - edited

          The v3 patch partially rejects because HBaseTestingUtility#createPreSplitLoadTestTable was added elsewhere. Looking into it, will commit if otherwise good.

          Edit: Never mind.

          Show
          Andrew Purtell added a comment - - edited The v3 patch partially rejects because HBaseTestingUtility#createPreSplitLoadTestTable was added elsewhere. Looking into it, will commit if otherwise good. Edit: Never mind.
          Hide
          stack added a comment -

          I'll cut a 0.92.2 RC after this and the upgrade to 1.0.3 goes in.

          Show
          stack added a comment - I'll cut a 0.92.2 RC after this and the upgrade to 1.0.3 goes in.
          Hide
          Lars Hofhansl added a comment -

          no objections here

          Show
          Lars Hofhansl added a comment - no objections here
          Hide
          Andrew Purtell added a comment -

          I'm going to commit this and then HBASE-6021 shortly if there are no objections.

          Show
          Andrew Purtell added a comment - I'm going to commit this and then HBASE-6021 shortly if there are no objections.
          Hide
          Andrew Purtell added a comment -

          This should go in before HBASE-6021 is committed, then that trivial patch can be committed on all branches.

          Show
          Andrew Purtell added a comment - This should go in before HBASE-6021 is committed, then that trivial patch can be committed on all branches.
          Hide
          Lars Hofhansl added a comment -

          LoadTestTool is in 0.94 and 0.96. So just targeting this against 0.92.

          Show
          Lars Hofhansl added a comment - LoadTestTool is in 0.94 and 0.96. So just targeting this against 0.92.
          Hide
          Devaraj Das added a comment -

          I ran this test on a 0.92 cluster. I tried the command

          hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4 -read 100:10 -bloom ROWCOL -key_window 5 -max_read_errors 10000 -num_keys 10000000000 -start_key 0

          Seems to be running well (its running for a few minutes now). Can we have this patch in the next 0.92 release, please [no significant changes to the core code too].

          Show
          Devaraj Das added a comment - I ran this test on a 0.92 cluster. I tried the command hbase org.apache.hadoop.hbase.util.LoadTestTool -write 50:100:20 -tn loadtest4 -read 100:10 -bloom ROWCOL -key_window 5 -max_read_errors 10000 -num_keys 10000000000 -start_key 0 Seems to be running well (its running for a few minutes now). Can we have this patch in the next 0.92 release, please [no significant changes to the core code too] .
          Hide
          Lars Hofhansl added a comment -

          Moved to 0.94.1.

          Show
          Lars Hofhansl added a comment - Moved to 0.94.1.
          Hide
          Jean-Daniel Cryans added a comment -

          The new code should be forward-ported, also the zk cluster key really should be using the format that ZKUtil.getZooKeeperClusterKey wants and it should be used else we'll have as many ways to point to a cluster as the number of tools we have.

          Show
          Jean-Daniel Cryans added a comment - The new code should be forward-ported, also the zk cluster key really should be using the format that ZKUtil.getZooKeeperClusterKey wants and it should be used else we'll have as many ways to point to a cluster as the number of tools we have.
          Hide
          Ted Yu added a comment -

          Patch v3 adds support for specifying parent zk node (suggested by J-D).

          Show
          Ted Yu added a comment - Patch v3 adds support for specifying parent zk node (suggested by J-D).
          Hide
          Ted Yu added a comment -

          Tried patch v2 on a 5 node cluster running latest 0.92 code:

          -zk 10.253.68.161 -zk_port 2181 -tn loadtool -num_keys 100 -write 3:120
          

          The result is fine:

          2012-01-05 15:00:01,732 DEBUG [HBaseWriterThread_14] client.MetaScanner(188): Scanning .META. starting at row=loadtool,3333332a,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@477a1767
          2012-01-05 15:00:01,869 DEBUG [HBaseWriterThread_13] client.MetaScanner(188): Scanning .META. starting at row=loadtool,3851eb7b,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@477a1767
          Failed to write keys: 0
          
          Show
          Ted Yu added a comment - Tried patch v2 on a 5 node cluster running latest 0.92 code: -zk 10.253.68.161 -zk_port 2181 -tn loadtool -num_keys 100 -write 3:120 The result is fine: 2012-01-05 15:00:01,732 DEBUG [HBaseWriterThread_14] client.MetaScanner(188): Scanning .META. starting at row=loadtool,3333332a,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@477a1767 2012-01-05 15:00:01,869 DEBUG [HBaseWriterThread_13] client.MetaScanner(188): Scanning .META. starting at row=loadtool,3851eb7b,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@477a1767 Failed to write keys: 0
          Hide
          Jean-Daniel Cryans added a comment -

          You are right I confused the two.

          I don't have an opinion on HBASE-4908.

          Show
          Jean-Daniel Cryans added a comment - You are right I confused the two. I don't have an opinion on HBASE-4908 .
          Hide
          Ted Yu added a comment -

          I ran patch through 0.92 test suite:

          Tests run: 1076, Failures: 0, Errors: 0, Skipped: 8
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 2:06:52.944s
          
          Show
          Ted Yu added a comment - I ran patch through 0.92 test suite: Tests run: 1076, Failures: 0, Errors: 0, Skipped: 8 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:06:52.944s
          Hide
          Ted Yu added a comment -

          @J-D:
          Although HBASE-4907 hasn't been completed, HBASE-4908 is resolved.
          So I think this backport may still go into 0.92.0, considering the changes to HBaseTestingUtility are just addition of a new method.
          The possibility of this backport making 0.92 unstable is minimal.

          Show
          Ted Yu added a comment - @J-D: Although HBASE-4907 hasn't been completed, HBASE-4908 is resolved. So I think this backport may still go into 0.92.0, considering the changes to HBaseTestingUtility are just addition of a new method. The possibility of this backport making 0.92 unstable is minimal.
          Hide
          Ted Yu added a comment -

          Patch v2 adds new config parameter zk_port for specifying zookeeper client port.

          Show
          Ted Yu added a comment - Patch v2 adds new config parameter zk_port for specifying zookeeper client port.
          Hide
          Jean-Daniel Cryans added a comment -

          Since the original jira is far from being done, I think this should target 0.92.1.

          Show
          Jean-Daniel Cryans added a comment - Since the original jira is far from being done, I think this should target 0.92.1.
          Hide
          Ted Yu added a comment -

          The patch, v1

          Show
          Ted Yu added a comment - The patch, v1

            People

            • Assignee:
              Andrew Purtell
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development