Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: wal
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      Related to HDFS-3280 and the HBase WAL slowdown on 0.23+
      It would be nice to have a simple tool like HFilePerformanceEvaluation, ...
      to be able to check easily the HLog performance.

      1. verify.txt
        12 kB
        stack
      2. verify.txt
        12 kB
        stack
      3. HBASE-5792-v2.patch
        9 kB
        Matteo Bertozzi
      4. HBASE-5792-v1.patch
        9 kB
        Matteo Bertozzi
      5. HBASE-5792-v0.patch
        8 kB
        Matteo Bertozzi

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522681/HBASE-5792-v0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1526//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1526//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1526//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522681/HBASE-5792-v0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1526//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1526//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1526//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Interesting tool.

          +    region = HRegion.createHRegion(regionInfo, regionRootDir, getConf(), htd);
          

          I don't see where region is closed.
          There should be some cleanup method after the benchmarking is completed.

          Show
          Ted Yu added a comment - Interesting tool. + region = HRegion.createHRegion(regionInfo, regionRootDir, getConf(), htd); I don't see where region is closed. There should be some cleanup method after the benchmarking is completed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522711/HBASE-5792-v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.master.TestSplitLogManager

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1532//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1532//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1532//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522711/HBASE-5792-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1532//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1532//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1532//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Minor comments:

          import org.apache.hadoop.conf.Configuration;
          

          The above import was not used.

              for (Thread t : threads) t.join();
          

          join() may throw InterruptedException. Shall we catch it and proceed to the next Thread to be joined ?

          Show
          Ted Yu added a comment - Minor comments: import org.apache.hadoop.conf.Configuration; The above import was not used. for ( Thread t : threads) t.join(); join() may throw InterruptedException. Shall we catch it and proceed to the next Thread to be joined ?
          Hide
          Matteo Bertozzi added a comment -

          Don't know if it makes sense catch the InterruptException, since we're trying to test write performance with nThreads... if we lost one thread, the test doesn't reflect what we've asked for.

          Show
          Matteo Bertozzi added a comment - Don't know if it makes sense catch the InterruptException, since we're trying to test write performance with nThreads... if we lost one thread, the test doesn't reflect what we've asked for.
          Hide
          Ted Yu added a comment -

          I wonder if it makes sense to persist benchmark results to HBase.
          That may show us some trend w.r.t. HLog performance.

          Show
          Ted Yu added a comment - I wonder if it makes sense to persist benchmark results to HBase. That may show us some trend w.r.t. HLog performance.
          Hide
          Todd Lipcon added a comment -

          I wonder if it makes sense to persist benchmark results to HBase.

          Let's not scope creep. The point of this tool is to have a quick benchmark you can run without having to set up an HBase cluster. Outputting results on the console is good enough to parse and store in whatever database you want. My preference would probably be just appending to a text file for easy graphing from Hudson.

          Show
          Todd Lipcon added a comment - I wonder if it makes sense to persist benchmark results to HBase. Let's not scope creep. The point of this tool is to have a quick benchmark you can run without having to set up an HBase cluster. Outputting results on the console is good enough to parse and store in whatever database you want. My preference would probably be just appending to a text file for easy graphing from Hudson.
          Hide
          Jonathan Hsieh added a comment -

          When I ran a previous version of this I believe it wanted a hbase cluster up, we should make sure this works without that.

          Show
          Jonathan Hsieh added a comment - When I ran a previous version of this I believe it wanted a hbase cluster up, we should make sure this works without that.
          Hide
          Matteo Bertozzi added a comment -

          @Jonathan the previous version had a HBaseAdmin call at startup because the tool contains also a Put test, but the hbase up si not needed.

          Show
          Matteo Bertozzi added a comment - @Jonathan the previous version had a HBaseAdmin call at startup because the tool contains also a Put test, but the hbase up si not needed.
          Hide
          stack added a comment -

          This is great Matteo. We need this. Yeah, agree, this tool will have most value if it puts nothing but a lone region (and WAL).

          Few minors below:

          Missing annotatations on audience.

          Do you need these? IIRC, the default exists w/ need of definition:

          +  public HLogPerformanceEvaluation() {
          +  }
          

          You do it in another place at least too.

          No harm adding a bit of class doc on HLogPutBenchmark

          You don't want to use a command parser?

          Show
          stack added a comment - This is great Matteo. We need this. Yeah, agree, this tool will have most value if it puts nothing but a lone region (and WAL). Few minors below: Missing annotatations on audience. Do you need these? IIRC, the default exists w/ need of definition: + public HLogPerformanceEvaluation() { + } You do it in another place at least too. No harm adding a bit of class doc on HLogPutBenchmark You don't want to use a command parser?
          Hide
          stack added a comment -

          @Matteo NVM. I want to use this tool now so I'll take care of the above. Good stuff.

          Show
          stack added a comment - @Matteo NVM. I want to use this tool now so I'll take care of the above. Good stuff.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12522825/HBASE-5792-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1539//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1539//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1539//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522825/HBASE-5792-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1539//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1539//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1539//console This message is automatically generated.
          Hide
          stack added a comment -

          Matteo's patch only I moved the tool down under the wal package. Also added a verify step that will look at resultant WAL and verify we are writing in sequence. Oh, I also made the tool actually write a log (Was using default constructor on region which won't create a log – have to create the log external and pass it in)

          Show
          stack added a comment - Matteo's patch only I moved the tool down under the wal package. Also added a verify step that will look at resultant WAL and verify we are writing in sequence. Oh, I also made the tool actually write a log (Was using default constructor on region which won't create a log – have to create the log external and pass it in)
          Hide
          stack added a comment -

          Committed to trunk. Thanks for the patch Matteo.

          Show
          stack added a comment - Committed to trunk. Thanks for the patch Matteo.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2769 (See https://builds.apache.org/job/HBase-TRUNK/2769/)
          HBASE-5792 HLog Performance Evaluation Tool (Revision 1326793)
          HBASE-5792 HLog Performance Evaluation Tool (Revision 1326792)

          Result = SUCCESS
          stack :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java

          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2769 (See https://builds.apache.org/job/HBase-TRUNK/2769/ ) HBASE-5792 HLog Performance Evaluation Tool (Revision 1326793) HBASE-5792 HLog Performance Evaluation Tool (Revision 1326792) Result = SUCCESS stack : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Todd Lipcon added a comment -

          I just noticed we already have a "TestHLogBench" tool which appears pretty close to this... we should merge the two in some way or be clear on what the difference is.

          Show
          Todd Lipcon added a comment - I just noticed we already have a "TestHLogBench" tool which appears pretty close to this... we should merge the two in some way or be clear on what the difference is.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #173 (See https://builds.apache.org/job/HBase-TRUNK-security/173/)
          HBASE-5792 HLog Performance Evaluation Tool (Revision 1326793)
          HBASE-5792 HLog Performance Evaluation Tool (Revision 1326792)

          Result = FAILURE
          stack :
          Files :

          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java

          stack :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #173 (See https://builds.apache.org/job/HBase-TRUNK-security/173/ ) HBASE-5792 HLog Performance Evaluation Tool (Revision 1326793) HBASE-5792 HLog Performance Evaluation Tool (Revision 1326792) Result = FAILURE stack : Files : /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java stack : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          stack added a comment -

          @Todd Thanks. I removed TestHLogBench over in HBASE-5808. The new test does verify and actually writes a log which TestHLogBench does not.

          Show
          stack added a comment - @Todd Thanks. I removed TestHLogBench over in HBASE-5808 . The new test does verify and actually writes a log which TestHLogBench does not.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #15 (See https://builds.apache.org/job/HBase-0.94-security/15/)
          HBASE-5792 HLog Performance Evaluation Tool (Revision 1327668)

          Result = SUCCESS
          larsh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #15 (See https://builds.apache.org/job/HBase-0.94-security/15/ ) HBASE-5792 HLog Performance Evaluation Tool (Revision 1327668) Result = SUCCESS larsh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #129 (See https://builds.apache.org/job/HBase-0.94/129/)
          HBASE-5792 HLog Performance Evaluation Tool (Revision 1327668)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #129 (See https://builds.apache.org/job/HBase-0.94/129/ ) HBASE-5792 HLog Performance Evaluation Tool (Revision 1327668) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java

            People

            • Assignee:
              Matteo Bertozzi
              Reporter:
              Matteo Bertozzi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development