Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.3, 0.95.2
    • Fix Version/s: 0.95.0
    • Component/s: Performance, Scanners
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This JIRA adds the ability of doing parallel seek in StoreScanner. It is off by default.
      The following config parameter turns on this feature:
      hbase.storescanner.parallel.seek.enable

      hbase.storescanner.parallel.seek.threads controls the number of threads in thread pool which serves parallel seeking. Default is 10 threads.
      Show
      This JIRA adds the ability of doing parallel seek in StoreScanner. It is off by default. The following config parameter turns on this feature: hbase.storescanner.parallel.seek.enable hbase.storescanner.parallel.seek.threads controls the number of threads in thread pool which serves parallel seeking. Default is 10 threads.

      Description

      seems there's a potential improvable space before doing scanner.next:

      StoreScanner.java
          if (explicitColumnQuery && lazySeekEnabledGlobally) {
            for (KeyValueScanner scanner : scanners) {
              scanner.requestSeek(matcher.getStartKey(), false, true);
            }
          } else {
            for (KeyValueScanner scanner : scanners) {
              scanner.seek(matcher.getStartKey());
            }
          }
      

      we can do scanner.requestSeek or scanner.seek in parallel, instead of current serialization, to reduce latency for special case.

      Any ideas on it ? I'll have a try if the comments/suggestions are positive:)

      1. HBASE-7495-v9.txt
        17 kB
        Liang Xie
      2. HBASE-7495-v8.txt
        16 kB
        Liang Xie
      3. HBASE-7495-v7.txt
        10 kB
        Liang Xie
      4. HBASE-7495-v6.txt
        10 kB
        Liang Xie
      5. HBASE-7495-v5.txt
        8 kB
        Liang Xie
      6. HBASE-7495-v4.txt
        4 kB
        Liang Xie
      7. HBASE-7495-v4.txt
        4 kB
        Liang Xie
      8. HBASE-7495-v3.txt
        4 kB
        Liang Xie
      9. HBASE-7495-v2.txt
        5 kB
        Liang Xie
      10. HBASE-7495-v11.txt
        24 kB
        Liang Xie
      11. HBASE-7495-v10.txt
        19 kB
        Liang Xie
      12. HBASE-7495-0.94.txt
        14 kB
        Liang Xie
      13. HBASE-7495.txt
        2 kB
        Liang Xie
      14. HBASE-7495.txt
        4 kB
        Liang Xie
      15. HBASE-7495.txt
        4 kB
        Liang Xie
      16. 7495-v12.txt
        22 kB
        Ted Yu

        Activity

        Hide
        stack added a comment -

        Marking closed.

        Show
        stack added a comment - Marking closed.
        Hide
        Liang Xie added a comment -

        Sounds a good news, Lars Hofhansl
        yes, my expected improvement is against more than one storefile and low cache(both hbase block cache and os fs cache layer) hit ratio.

        Show
        Liang Xie added a comment - Sounds a good news, Lars Hofhansl yes, my expected improvement is against more than one storefile and low cache(both hbase block cache and os fs cache layer) hit ratio.
        Hide
        Lars Hofhansl added a comment -

        In my limited local testing with 0.94 patch I did not see any performance detriment with this patch (I tested with all data in the OS cache, and separately with all data in the block cache).

        In my scenario I also did not see any improvement, but I also do not have many store files, and only a single Datanode.

        Show
        Lars Hofhansl added a comment - In my limited local testing with 0.94 patch I did not see any performance detriment with this patch (I tested with all data in the OS cache, and separately with all data in the block cache). In my scenario I also did not see any improvement, but I also do not have many store files, and only a single Datanode.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #412 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/412/)
        HBASE-7495 parallel seek in StoreScanner (Liang Xie) (Revision 1447740)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/ParallelSeekHandler.java
        • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #412 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/412/ ) HBASE-7495 parallel seek in StoreScanner (Liang Xie) (Revision 1447740) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/ParallelSeekHandler.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3885 (See https://builds.apache.org/job/HBase-TRUNK/3885/)
        HBASE-7495 parallel seek in StoreScanner (Liang Xie) (Revision 1447740)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/ParallelSeekHandler.java
        • /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3885 (See https://builds.apache.org/job/HBase-TRUNK/3885/ ) HBASE-7495 parallel seek in StoreScanner (Liang Xie) (Revision 1447740) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestBulkDeleteProtocol.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/ParallelSeekHandler.java /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
        Hide
        Ted Yu added a comment -

        Integrated to trunk.

        Thanks for the patch, Liang.

        Thanks for the reviews, Chunhui, Sergey and Lars.

        Show
        Ted Yu added a comment - Integrated to trunk. Thanks for the patch, Liang. Thanks for the reviews, Chunhui, Sergey and Lars.
        Hide
        Liang Xie added a comment -

        Thanks for review, Lars Hofhansl
        i am OK with for trunk only right now. i'll do more work per shenchunhui's suggestion, especially, maybe i need to have a benchmark on hight block cache hit ratio scenario, before considering it for 0.94

        Show
        Liang Xie added a comment - Thanks for review, Lars Hofhansl i am OK with for trunk only right now. i'll do more work per shenchunhui 's suggestion, especially, maybe i need to have a benchmark on hight block cache hit ratio scenario, before considering it for 0.94
        Hide
        Lars Hofhansl added a comment -

        +1 for trunk.

        I'd like to have some more discussion before/if we consider this for 0.94.
        Any thoughts about that?

        Show
        Lars Hofhansl added a comment - +1 for trunk. I'd like to have some more discussion before/if we consider this for 0.94. Any thoughts about that?
        Hide
        Ted Yu added a comment -

        Will integrate this tomorrow morning if there is no further review comment.

        Show
        Ted Yu added a comment - Will integrate this tomorrow morning if there is no further review comment.
        Hide
        Liang Xie added a comment -

        Yes, the parallel-seeking feature is disabled by default in current patch.

        Show
        Liang Xie added a comment - Yes, the parallel-seeking feature is disabled by default in current patch.
        Hide
        Liang Xie added a comment -

        shenchunhui, yeh, good idea, i can't agree more.

        but to be honest,i'd like to do per-table or other granularity configue in another jira, to make parallel-seeking feature more configurable. it's just boring to rebase again(if conflicts with new patches) for me

        is it suitable to merge Ted's last rebased v12 patch into trunk now ?

        Thanks very much

        Show
        Liang Xie added a comment - shenchunhui , yeh, good idea, i can't agree more. but to be honest,i'd like to do per-table or other granularity configue in another jira, to make parallel-seeking feature more configurable. it's just boring to rebase again(if conflicts with new patches) for me is it suitable to merge Ted's last rebased v12 patch into trunk now ? Thanks very much
        Hide
        chunhui shen added a comment -

        From the above test results, I think this feature is useful for the case where ops is low and latency is high.
        But will it become worse under some cases? e.g. thousands of ops with some block cache hit ratio
        Since we couldn't make it as a default confiuration, how about per-table or per-family configuration? It may be better to enable parallel seeking

        Show
        chunhui shen added a comment - From the above test results, I think this feature is useful for the case where ops is low and latency is high. But will it become worse under some cases? e.g. thousands of ops with some block cache hit ratio Since we couldn't make it as a default confiuration, how about per-table or per-family configuration? It may be better to enable parallel seeking
        Hide
        Ted Yu added a comment -

        I ran TestSplitTransactionOnCluster locally and it passed.

        Show
        Ted Yu added a comment - I ran TestSplitTransactionOnCluster locally and it passed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569677/7495-v12.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569677/7495-v12.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4449//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12569675/7495-v12.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12569675/7495-v12.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4448//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Patch rebased on trunk.

        Show
        Ted Yu added a comment - Patch rebased on trunk.
        Hide
        Ted Yu added a comment -

        The results are encouraging.

        +1.

        Show
        Ted Yu added a comment - The results are encouraging. +1.
        Hide
        Liang Xie added a comment -

        Here is a test report with different thread num.
        Test Env&Data:
        0.94.3 + CDH4.1
        one billion rows, 10 RegionServer&Datanode, 179 storefiles in total, each with 2T*12 SATA disk,
        hfile.block.cache.size=0.25, 10 regions in each RS,most of numberOfStorefiles in each region are between 2 and 3(similar with 179/100)

        dfs -du -s -h /hbase/lgxl-xieliang/YCSBTest/
        605.3g /hbase/lgxl-xieliang/YCSBTest

        Test Result

        Original:

        threads Throughput(ops/sec) Operations AverageLatency(us) 50thPercentileLatency(ms) 95thPercentileLatency(ms) 99thPercentileLatency(ms)
        1 37.5947869273203 11279 26579.78154091675 23 46 87
        5 205.5726831869615 61679 24315.654809578624 19 52 65
        10 406.97476779102357 122114 24564.425463091866 19 50 62

        Patched:

        threads Throughput(ops/sec) Operations AverageLatency(us) 50thPercentileLatency(ms) 95thPercentileLatency(ms) 99thPercentileLatency(ms)
        1 64.66311778960737 19399 15455.213516160627 13 25 55
        5 288.76356558142015 86636 17309.12692183388 11 44 53
        10 560.8241857117149 168291 24564.425463091866 12 42 52
        Show
        Liang Xie added a comment - Here is a test report with different thread num. Test Env&Data: 0.94.3 + CDH4.1 one billion rows, 10 RegionServer&Datanode, 179 storefiles in total, each with 2T*12 SATA disk, hfile.block.cache.size=0.25, 10 regions in each RS,most of numberOfStorefiles in each region are between 2 and 3(similar with 179/100) dfs -du -s -h /hbase/lgxl-xieliang/YCSBTest/ 605.3g /hbase/lgxl-xieliang/YCSBTest Test Result Original: threads Throughput(ops/sec) Operations AverageLatency(us) 50thPercentileLatency(ms) 95thPercentileLatency(ms) 99thPercentileLatency(ms) 1 37.5947869273203 11279 26579.78154091675 23 46 87 5 205.5726831869615 61679 24315.654809578624 19 52 65 10 406.97476779102357 122114 24564.425463091866 19 50 62 Patched: threads Throughput(ops/sec) Operations AverageLatency(us) 50thPercentileLatency(ms) 95thPercentileLatency(ms) 99thPercentileLatency(ms) 1 64.66311778960737 19399 15455.213516160627 13 25 55 5 288.76356558142015 86636 17309.12692183388 11 44 53 10 560.8241857117149 168291 24564.425463091866 12 42 52
        Hide
        Ted Yu added a comment -

        Review request is at https://reviews.apache.org/r/9272/

        The new TestCoprocessorScanPolicy ctor would be called by JUnit since TestCoprocessorScanPolicy is parameterized.

        Show
        Ted Yu added a comment - Review request is at https://reviews.apache.org/r/9272/ The new TestCoprocessorScanPolicy ctor would be called by JUnit since TestCoprocessorScanPolicy is parameterized.
        Hide
        Sergey Shelukhin added a comment -

        latest patch looks reasonable but what about the tests? TestCoprocessorScanPolicy has a ctor that enables new feature, but I don't see it called.

        Show
        Sergey Shelukhin added a comment - latest patch looks reasonable but what about the tests? TestCoprocessorScanPolicy has a ctor that enables new feature, but I don't see it called.
        Hide
        Liang Xie added a comment -

        the line which size longer than 100 is old code:

        M_META_SERVER_SHUTDOWN    (72, ExecutorType.MASTER_META_SERVER_OPERATIONS),  // Master is processing shutdown of RS hosting a meta region (-ROOT- or .META.).
        

        I just changed semicolon to comma :-P

        Show
        Liang Xie added a comment - the line which size longer than 100 is old code: M_META_SERVER_SHUTDOWN (72, ExecutorType.MASTER_META_SERVER_OPERATIONS), // Master is processing shutdown of RS hosting a meta region (-ROOT- or .META.). I just changed semicolon to comma :-P
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567828/HBASE-7495-v11.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces lines longer than 100

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567828/HBASE-7495-v11.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4315//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        I reran TestHLog with latest v11 patch on my box, it's OK:
        Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
        Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.697 sec

        Results :

        Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

        Show
        Liang Xie added a comment - I reran TestHLog with latest v11 patch on my box, it's OK: Running org.apache.hadoop.hbase.regionserver.wal.TestHLog Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.697 sec Results : Tests run: 11, Failures: 0, Errors: 0, Skipped: 0
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567823/HBASE-7495-v11.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.wal.TestHLog

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567823/HBASE-7495-v11.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestHLog Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4314//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        v11 patch integrated into HBase's ExecutorService

        Show
        Liang Xie added a comment - v11 patch integrated into HBase's ExecutorService
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567725/HBASE-7495-v10.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 15 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567725/HBASE-7495-v10.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 15 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4305//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567726/HBASE-7495-0.94.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4306//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567726/HBASE-7495-0.94.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4306//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        chunhui shen, I've uploaded a patch for 0.94, please go ahead, thanks very much

        Show
        Liang Xie added a comment - chunhui shen , I've uploaded a patch for 0.94, please go ahead, thanks very much
        Hide
        Liang Xie added a comment -

        v10 adds a method in RegionServerServices per Ted's comments

        Show
        Liang Xie added a comment - v10 adds a method in RegionServerServices per Ted's comments
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567717/HBASE-7495-v9.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 1 zombie test(s):

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567717/HBASE-7495-v9.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4303//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Thanks for your persistence.

        +      boolean parallelSeekConfFlag = rsService.getConfiguration().getBoolean(
        +        STORESCANNER_PARALLEL_SEEK_ENABLE, false);
        +      if (!parallelSeekConfFlag || !(rsService instanceof HRegionServer)) return;
        

        My comment @ 31/Jan/13 14:32 was for adding getParallelSFSeekExecutor() in RegionServerServices.

        But this is minor.

        Show
        Ted Yu added a comment - Thanks for your persistence. + boolean parallelSeekConfFlag = rsService.getConfiguration().getBoolean( + STORESCANNER_PARALLEL_SEEK_ENABLE, false ); + if (!parallelSeekConfFlag || !(rsService instanceof HRegionServer)) return ; My comment @ 31/Jan/13 14:32 was for adding getParallelSFSeekExecutor() in RegionServerServices. But this is minor.
        Hide
        Liang Xie added a comment -

        addressed above coments. about thread number, IMHO, it's not very necessary to set for CachedThreadPool, please correct me if wrong

        Show
        Liang Xie added a comment - addressed above coments. about thread number, IMHO, it's not very necessary to set for CachedThreadPool, please correct me if wrong
        Hide
        chunhui shen added a comment -

        Liang Xie
        Could you upload a patch for 0.94.

        I'm interesting to test the performance under our benchmark.

        We wonder whether it will be worse in a normal case. e.g. much more concurrent threads, with a cache hit ratio, normal storefiles( like 3) in the region.

        "Disabled block cache in RS,numberOfStorefiles in each region are around 40,1 thread"
        These three conditions seem not actual in real production system. Or could you take a simple description of application about this scenario.

        Thanks

        Show
        chunhui shen added a comment - Liang Xie Could you upload a patch for 0.94. I'm interesting to test the performance under our benchmark. We wonder whether it will be worse in a normal case. e.g. much more concurrent threads, with a cache hit ratio, normal storefiles( like 3) in the region. "Disabled block cache in RS,numberOfStorefiles in each region are around 40,1 thread" These three conditions seem not actual in real production system. Or could you take a simple description of application about this scenario. Thanks
        Hide
        Ted Yu added a comment -
        +    } catch (ExecutionException e) {
        +      throw new InterruptedIOException(e.getMessage());
        +    } catch (CancellationException ce) {
        +      throw new InterruptedIOException(ce.getMessage());
        

        Throwing IOException would be appropriate.

        Show
        Ted Yu added a comment - + } catch (ExecutionException e) { + throw new InterruptedIOException(e.getMessage()); + } catch (CancellationException ce) { + throw new InterruptedIOException(ce.getMessage()); Throwing IOException would be appropriate.
        Hide
        Ted Yu added a comment -

        Putting patch on review board would allow people to make review comments more effectively.

        +  public ThreadPoolExecutor getParallelSFSeekExecutor() {
        

        The above method can be package private.

        +  private boolean parallelSFSeekEnabled;
        +  private ThreadPoolExecutor parallelSFSeekExecutor;
        

        Do you need the boolean flag ? parallelSFSeekExecutor != null would mean parallel seek enabled.

        +  public static final String STORESCANNER_PARALLEL_SEEK_ENABLE =
        +      "hbase.storescanner.parallel.seek.enable";
        

        From my comment above, you can see other ExecutorService specifies number of threads. Please consider replacing the above config param with "hbase.storescanner.parallel.seek.threads".

        Show
        Ted Yu added a comment - Putting patch on review board would allow people to make review comments more effectively. + public ThreadPoolExecutor getParallelSFSeekExecutor() { The above method can be package private. + private boolean parallelSFSeekEnabled; + private ThreadPoolExecutor parallelSFSeekExecutor; Do you need the boolean flag ? parallelSFSeekExecutor != null would mean parallel seek enabled. + public static final String STORESCANNER_PARALLEL_SEEK_ENABLE = + "hbase.storescanner.parallel.seek.enable" ; From my comment above, you can see other ExecutorService specifies number of threads. Please consider replacing the above config param with "hbase.storescanner.parallel.seek.threads".
        Hide
        Ted Yu added a comment -

        Glad to see latency come down for parallel seek.

        w.r.t. changes in TestCoprocessorScanPolicy, you can use this annotation:

        @RunWith(Parameterized.class)
        

        And specify parameters with:

          @Parameters
          public static Collection<Object[]> parameters() {
        

        Take a look at TestMiniClusterLoadSequential.
        This is how HRegionServer starts ExecutorService:

            this.service.startExecutorService(ExecutorType.RS_OPEN_REGION,
              conf.getInt("hbase.regionserver.executor.openregion.threads", 3));
        

        Do you think adding another ExecutorType enum and using the above pattern is better ?

        Show
        Ted Yu added a comment - Glad to see latency come down for parallel seek. w.r.t. changes in TestCoprocessorScanPolicy, you can use this annotation: @RunWith(Parameterized.class) And specify parameters with: @Parameters public static Collection< Object []> parameters() { Take a look at TestMiniClusterLoadSequential. This is how HRegionServer starts ExecutorService: this .service.startExecutorService(ExecutorType.RS_OPEN_REGION, conf.getInt( "hbase.regionserver.executor.openregion.threads" , 3)); Do you think adding another ExecutorType enum and using the above pattern is better ?
        Hide
        Liang Xie added a comment -

        And i want to clarify again: if we have only one storefile or the target block is cached in OS/HBase block cache layer, then no latency benefit will be observed; if we have a total IO-bound env(e.g. every datanode IO %util is around 100%), i guess there's no obvious improvement as well.

        Show
        Liang Xie added a comment - And i want to clarify again: if we have only one storefile or the target block is cached in OS/HBase block cache layer, then no latency benefit will be observed; if we have a total IO-bound env(e.g. every datanode IO %util is around 100%), i guess there's no obvious improvement as well.
        Hide
        ramkrishna.s.vasudevan added a comment -

        Appreciate your persistance on this patch. Figures shows good improvement. Nice.

        Show
        ramkrishna.s.vasudevan added a comment - Appreciate your persistance on this patch. Figures shows good improvement. Nice.
        Hide
        Liang Xie added a comment -

        The failed case seems not related with this patch, and i ran it on my box and passed.
        About testing in real cluster, i did it today after applied v8 patch into 0.94.3:

        $./hdfs dfs -du -s -h hdfs://lgxl-xieliang/
        619.0g hdfs://lgxl-xieliang/

        one billion rows, 10 RegionServer&Datanode, each with 2T*12 SATA disk. Disabled block cache in RS, 10 regions in each RS, most of numberOfStorefiles in each region are around 40, and most of single storefileSizes are around 160M+. I ran with 1 thread in YCSB:

        With original 0.94.3:

        2013-02-01 14:33:39:563 300 sec: 570 operations; 2.3 current ops/sec; [READ AverageLatency(us)=419965.74]
        2013-02-01 14:33:39:599 300 sec: 571 operations; 27.78 current ops/sec; [READ AverageLatency(us)=616807]
        [OVERALL], RunTime(ms), 300051.0
        [OVERALL], Throughput(ops/sec), 1.9030098216636504
        [READ], Operations, 571
        [READ], AverageLatency(us), 525441.6672504378
        [READ], MinLatency(us), 78542
        [READ], MaxLatency(us), 1481186
        [READ], 50thPercentileLatency(ms), 448
        [READ], 95thPercentileLatency(ms), 854
        [READ], Return=0, 571

        Applied v8 patch, hbase.storescanner.parallel.seek.enable = false :

        2013-02-01 16:49:27:624 300 sec: 643 operations; 2.5 current ops/sec; [READ AverageLatency(us)=401727.8]
        2013-02-01 16:49:28:032 300 sec: 644 operations; 2.45 current ops/sec; [READ AverageLatency(us)=474225]
        [OVERALL], RunTime(ms), 300424.0
        [OVERALL], Throughput(ops/sec), 2.143636993049823
        [READ], Operations, 644
        [READ], AverageLatency(us), 466452.77639751555
        [READ], MinLatency(us), 156654
        [READ], MaxLatency(us), 1279945
        [READ], 50thPercentileLatency(ms), 439
        [READ], 95thPercentileLatency(ms), 724
        [READ], Return=0, 644

        Applied v8 patch, hbase.storescanner.parallel.seek.enable = true :

        2013-02-01 16:59:34:689 300 sec: 4594 operations; 16.01 current ops/sec; [READ AverageLatency(us)=63779.27]
        [OVERALL], RunTime(ms), 300008.0
        [OVERALL], Throughput(ops/sec), 15.31292498866697
        [READ], Operations, 4594
        [READ], AverageLatency(us), 65277.21745755333
        [READ], MinLatency(us), 7661
        [READ], MaxLatency(us), 1026107
        [READ], 50thPercentileLatency(ms), 52
        [READ], 95thPercentileLatency(ms), 129
        [READ], 99thPercentileLatency(ms), 272
        [READ], Return=0, 4594

        Show
        Liang Xie added a comment - The failed case seems not related with this patch, and i ran it on my box and passed. About testing in real cluster, i did it today after applied v8 patch into 0.94.3: $./hdfs dfs -du -s -h hdfs://lgxl-xieliang/ 619.0g hdfs://lgxl-xieliang/ one billion rows, 10 RegionServer&Datanode, each with 2T*12 SATA disk. Disabled block cache in RS, 10 regions in each RS, most of numberOfStorefiles in each region are around 40, and most of single storefileSizes are around 160M+. I ran with 1 thread in YCSB: With original 0.94.3: 2013-02-01 14:33:39:563 300 sec: 570 operations; 2.3 current ops/sec; [READ AverageLatency(us)=419965.74] 2013-02-01 14:33:39:599 300 sec: 571 operations; 27.78 current ops/sec; [READ AverageLatency(us)=616807] [OVERALL] , RunTime(ms), 300051.0 [OVERALL] , Throughput(ops/sec), 1.9030098216636504 [READ] , Operations, 571 [READ] , AverageLatency(us), 525441.6672504378 [READ] , MinLatency(us), 78542 [READ] , MaxLatency(us), 1481186 [READ] , 50thPercentileLatency(ms), 448 [READ] , 95thPercentileLatency(ms), 854 [READ] , Return=0, 571 Applied v8 patch, hbase.storescanner.parallel.seek.enable = false : 2013-02-01 16:49:27:624 300 sec: 643 operations; 2.5 current ops/sec; [READ AverageLatency(us)=401727.8] 2013-02-01 16:49:28:032 300 sec: 644 operations; 2.45 current ops/sec; [READ AverageLatency(us)=474225] [OVERALL] , RunTime(ms), 300424.0 [OVERALL] , Throughput(ops/sec), 2.143636993049823 [READ] , Operations, 644 [READ] , AverageLatency(us), 466452.77639751555 [READ] , MinLatency(us), 156654 [READ] , MaxLatency(us), 1279945 [READ] , 50thPercentileLatency(ms), 439 [READ] , 95thPercentileLatency(ms), 724 [READ] , Return=0, 644 Applied v8 patch, hbase.storescanner.parallel.seek.enable = true : 2013-02-01 16:59:34:689 300 sec: 4594 operations; 16.01 current ops/sec; [READ AverageLatency(us)=63779.27] [OVERALL] , RunTime(ms), 300008.0 [OVERALL] , Throughput(ops/sec), 15.31292498866697 [READ] , Operations, 4594 [READ] , AverageLatency(us), 65277.21745755333 [READ] , MinLatency(us), 7661 [READ] , MaxLatency(us), 1026107 [READ] , 50thPercentileLatency(ms), 52 [READ] , 95thPercentileLatency(ms), 129 [READ] , 99thPercentileLatency(ms), 272 [READ] , Return=0, 4594
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567544/HBASE-7495-v8.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567544/HBASE-7495-v8.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4289//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567541/HBASE-7495-v8.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4288//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567541/HBASE-7495-v8.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4288//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Move executor to HRegionService now in v8

        Show
        Liang Xie added a comment - Move executor to HRegionService now in v8
        Hide
        Ted Yu added a comment -

        I don't think 22 more seconds in QA run are noticeable

        Can you put the ExecutorService in the right place and perform testing on real cluster ?

        Thanks

        Show
        Ted Yu added a comment - I don't think 22 more seconds in QA run are noticeable Can you put the ExecutorService in the right place and perform testing on real cluster ? Thanks
        Hide
        Liang Xie added a comment -

        Hi Ted Yu, i ran the TestCoprocessorScanPolicy case, it costs 22+s in my box, if we have two cases w/ and w/o StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE being true, it means the whole daily QA running time will add 22+s.
        Is it ok that add "conf.setBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, true);" into setUpBeforeClass(). It'll always enable this feature in TestCoprocessorScanPolicy case, without making QA time longger ?

        Show
        Liang Xie added a comment - Hi Ted Yu , i ran the TestCoprocessorScanPolicy case, it costs 22+s in my box, if we have two cases w/ and w/o StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE being true, it means the whole daily QA running time will add 22+s. Is it ok that add "conf.setBoolean(StoreScanner.STORESCANNER_PARALLEL_SEEK_ENABLE, true);" into setUpBeforeClass(). It'll always enable this feature in TestCoprocessorScanPolicy case, without making QA time longger ?
        Hide
        Ted Yu added a comment -

        Can you make TestCoprocessorScanPolicy parameterized test so that "hbase.storescanner.parallel.seek.enable" being true can be exercised ?

        Show
        Ted Yu added a comment - Can you make TestCoprocessorScanPolicy parameterized test so that "hbase.storescanner.parallel.seek.enable" being true can be exercised ?
        Hide
        Ted Yu added a comment -

        store.getHRegion() returns the region which has rsServices field.
        You can create a package private getter in HRegion so that RegionServerServices can be accessed.

        Show
        Ted Yu added a comment - store.getHRegion() returns the region which has rsServices field. You can create a package private getter in HRegion so that RegionServerServices can be accessed.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567341/HBASE-7495-v7.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567341/HBASE-7495-v7.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4273//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567339/HBASE-7495-v6.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4272//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567339/HBASE-7495-v6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4272//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567324/HBASE-7495-v6.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567324/HBASE-7495-v6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4271//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Anoop Sam John, good point, it's a trash code came from previous version, let me remove it from the v6 file now.

        Show
        Liang Xie added a comment - Anoop Sam John , good point, it's a trash code came from previous version, let me remove it from the v6 file now.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567321/HBASE-7495-v6.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567321/HBASE-7495-v6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4270//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Thanks my colleague Honghua Feng for the MVCC code improvement

        Show
        Liang Xie added a comment - Thanks my colleague Honghua Feng for the MVCC code improvement
        Hide
        Liang Xie added a comment -

        Uploaded patch v6, move the MVCC setThreadReadPoint from ScannerSeekWorker constructor into call block
        For passing config object, seems it need to add a new param in StoreScanner's constructor, and probably need to repair many broken test cases? In my patch, it's initialized just one time in static block, i think it's fine as well.
        Ted Yu, if we move the ExecutorService to HRegionServer class, we need to expose it with static getter, right? since in StoreScanner class, we could not get the current HRegionServer instance easily in current codebase. but if we add a static getter method, it'll bring several FindBug warnings.

        Show
        Liang Xie added a comment - Uploaded patch v6, move the MVCC setThreadReadPoint from ScannerSeekWorker constructor into call block For passing config object, seems it need to add a new param in StoreScanner's constructor, and probably need to repair many broken test cases? In my patch, it's initialized just one time in static block, i think it's fine as well. Ted Yu , if we move the ExecutorService to HRegionServer class, we need to expose it with static getter, right? since in StoreScanner class, we could not get the current HRegionServer instance easily in current codebase. but if we add a static getter method, it'll bring several FindBug warnings.
        Hide
        Anoop Sam John added a comment -
        +    List<Callable<Void>> tasks = new ArrayList<Callable<Void>>(storeFileScannerCount);
        

        Why we need this list?

        Show
        Anoop Sam John added a comment - + List<Callable< Void >> tasks = new ArrayList<Callable< Void >>(storeFileScannerCount); Why we need this list?
        Hide
        Sergey Shelukhin added a comment -
        +  static final boolean parallelSeek = HBaseConfiguration.create().
        +      getBoolean(HConstants.STORESCANNER_SEEK_PARALLEL_ENABLED, false);
        

        The config object should be passed from outside... Creating it is unnecessary (and can cause inconsistency)

        LOG.error("");
        

        Exception should at least be logged.

        Show
        Sergey Shelukhin added a comment - + static final boolean parallelSeek = HBaseConfiguration.create(). + getBoolean(HConstants.STORESCANNER_SEEK_PARALLEL_ENABLED, false ); The config object should be passed from outside... Creating it is unnecessary (and can cause inconsistency) LOG.error(""); Exception should at least be logged.
        Hide
        Ted Yu added a comment -

        In patch v5, the ExecutorService is created in StoreScanner.
        I think it should belong to region server.

        Show
        Ted Yu added a comment - In patch v5, the ExecutorService is created in StoreScanner. I think it should belong to region server.
        Hide
        Ted Yu added a comment -
        +   * A flag that enables StoreFileScanner parallel-seeking in StoreScanner
        +   */
        +  public static final String STORESCANNER_SEEK_PARALLEL_ENABLED =
        +      "hbase.storescanner.seek.parallel.enabled";
        

        The constant is only used in StoreScanner.java
        Please move it there. Maybe name it "hbase.storescanner.parallel.seek.enable"

        +   * seek storefile in parallel to optimize IO latency as much as possible
        +   * @throws IOException
        +   */
        +  private void parallelSeek(final List<? extends KeyValueScanner>
        +      scanners, final KeyValue keyValue) throws IOException {
        

        'seek storefile' -> 'seek storefiles'
        Please add javadoc for parameters.

        +      List<Future<Void>> futures = seekExecutor.invokeAll(tasks);
        

        Why not call seekExecutor.submit(task) in the loop ?

        +    } catch (InterruptedException ie) {
        +      LOG.error("");
        +      throw new IOException(ie);
        

        Please use InterruptedIOException. If you don't have custom error log, you can omit LOG.error() statement.

        Show
        Ted Yu added a comment - + * A flag that enables StoreFileScanner parallel-seeking in StoreScanner + */ + public static final String STORESCANNER_SEEK_PARALLEL_ENABLED = + "hbase.storescanner.seek.parallel.enabled" ; The constant is only used in StoreScanner.java Please move it there. Maybe name it "hbase.storescanner.parallel.seek.enable" + * seek storefile in parallel to optimize IO latency as much as possible + * @ throws IOException + */ + private void parallelSeek( final List<? extends KeyValueScanner> + scanners, final KeyValue keyValue) throws IOException { 'seek storefile' -> 'seek storefiles' Please add javadoc for parameters. + List<Future< Void >> futures = seekExecutor.invokeAll(tasks); Why not call seekExecutor.submit(task) in the loop ? + } catch (InterruptedException ie) { + LOG.error(""); + throw new IOException(ie); Please use InterruptedIOException. If you don't have custom error log, you can omit LOG.error() statement.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12567132/HBASE-7495-v5.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 1 zombie test(s):

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12567132/HBASE-7495-v5.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4249//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Attached v5 patch introduced a config key to enable/disable the parallel-seeking feature, and it's off by default.

        Show
        Liang Xie added a comment - Attached v5 patch introduced a config key to enable/disable the parallel-seeking feature, and it's off by default.
        Hide
        Enis Soztutar added a comment -

        Maybe use ExecutorService to reduce thread production.

        Agreed. Starting a thread per seek would still be costly.

        Show
        Enis Soztutar added a comment - Maybe use ExecutorService to reduce thread production. Agreed. Starting a thread per seek would still be costly.
        Hide
        Ted Yu added a comment -

        and thread production

        Maybe use ExecutorService to reduce thread production.

        Show
        Ted Yu added a comment - and thread production Maybe use ExecutorService to reduce thread production.
        Hide
        chunhui shen added a comment -

        It's better to measure the performance in the normal case with some pressure.

        Parallel seeking seems to decrease response time, but in fact we would take resource for context switch frequently and thread production

        Show
        chunhui shen added a comment - It's better to measure the performance in the normal case with some pressure. Parallel seeking seems to decrease response time, but in fact we would take resource for context switch frequently and thread production
        Hide
        Matteo Bertozzi added a comment -

        As Ted suggested, can you measure the performance gain in a real cluster?
        especially when there're multiple scan request on one and more tables
        having a table like numConcurrentScan/numTables with the performance numbers it will be nice.

        Show
        Matteo Bertozzi added a comment - As Ted suggested, can you measure the performance gain in a real cluster? especially when there're multiple scan request on one and more tables having a table like numConcurrentScan/numTables with the performance numbers it will be nice.
        Hide
        Sergey Shelukhin added a comment -

        +1 on latest patch after some testing...

        Show
        Sergey Shelukhin added a comment - +1 on latest patch after some testing...
        Hide
        Ted Yu added a comment -

        In my opinion, one loop should be enough for the task.
        You can create latch with initial count of scanners.size(). In the loop, after finding StoreFileScanner, a worker is created and started. We still maintain storeFileScannerCount. For non-StoreFileScanner, seek() is called.
        Outside the loop, first count down the latch by scanners.size()-storeFileScannerCount. Then wait for the latch.

        If you prefer, you can extract calling non-StoreFileScanner's seek() in a second loop.

        Show
        Ted Yu added a comment - In my opinion, one loop should be enough for the task. You can create latch with initial count of scanners.size(). In the loop, after finding StoreFileScanner, a worker is created and started. We still maintain storeFileScannerCount. For non-StoreFileScanner, seek() is called. Outside the loop, first count down the latch by scanners.size()-storeFileScannerCount. Then wait for the latch. If you prefer, you can extract calling non-StoreFileScanner's seek() in a second loop.
        Hide
        Ted Yu added a comment - - edited
        +   * Do StoreFileScanner.seek() in parallel
        +   * @throws IOException
        +   */
        +  private void parallelSeek(final List<? extends KeyValueScanner>
        +      scanners, final KeyValue keyValue) throws IOException {
        

        Add javadoc for the new method.

        +      if (scanner instanceof StoreFileScanner) {
        +        storeFileScannerNum++;
        

        storeFileScannerNum -> storeFileScannerCount

        +    for (ScannerSeekWorker worker : workers) {
        +      worker.start();
        +    }
        

        Why use a separate loop to start the workers ?

        +    } catch (InterruptedException e) {
        +      LOG.error("", e);
        +    }
        

        Restore interrupt status or throw InterruptedIOException.

        +    for (ScannerSeekWorker worker : workers) {
        +      if (worker.getErr() != null) {
        +        throw new IOException(worker.getErr());
        +      }
        +    }
        

        Use MultipleIOException so that user knows about more than one error.

        +      } catch (IOException e) {
        +        LOG.info("", e);
        

        Change to error level.

        Show
        Ted Yu added a comment - - edited + * Do StoreFileScanner.seek() in parallel + * @ throws IOException + */ + private void parallelSeek( final List<? extends KeyValueScanner> + scanners, final KeyValue keyValue) throws IOException { Add javadoc for the new method. + if (scanner instanceof StoreFileScanner) { + storeFileScannerNum++; storeFileScannerNum -> storeFileScannerCount + for (ScannerSeekWorker worker : workers) { + worker.start(); + } Why use a separate loop to start the workers ? + } catch (InterruptedException e) { + LOG.error("", e); + } Restore interrupt status or throw InterruptedIOException. + for (ScannerSeekWorker worker : workers) { + if (worker.getErr() != null ) { + throw new IOException(worker.getErr()); + } + } Use MultipleIOException so that user knows about more than one error. + } catch (IOException e) { + LOG.info("", e); Change to error level.
        Hide
        Ted Yu added a comment -

        @Liang:
        Can you measure the performance gain in a real cluster ?

        Thanks

        Show
        Ted Yu added a comment - @Liang: Can you measure the performance gain in a real cluster ? Thanks
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566940/HBASE-7495-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566940/HBASE-7495-v4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4233//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        The previous v4 patch is not right, i've uploaded again.
        And, i ran the IntegrationTestLazyCfLoading IT case and passed:
        Running org.apache.hadoop.hbase.IntegrationTestLazyCfLoading
        Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.77 sec

        Show
        Liang Xie added a comment - The previous v4 patch is not right, i've uploaded again. And, i ran the IntegrationTestLazyCfLoading IT case and passed: Running org.apache.hadoop.hbase.IntegrationTestLazyCfLoading Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.77 sec
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566922/HBASE-7495-v4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:

        -1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.filter.TestFilter.testFilterListWithSingleColumnValueFilter(TestFilter.java:1193)

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566922/HBASE-7495-v4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: -1 core zombie tests . There are 1 zombie test(s): at org.apache.hadoop.hbase.filter.TestFilter.testFilterListWithSingleColumnValueFilter(TestFilter.java:1193) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4232//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Thanks Sergey Shelukhin for looking into it, attached v4 patch should address your comment
        I'll run the above mentioned IT for scans asap

        Show
        Liang Xie added a comment - Thanks Sergey Shelukhin for looking into it, attached v4 patch should address your comment I'll run the above mentioned IT for scans asap
        Hide
        Sergey Shelukhin added a comment -

        Also, there's an integration test for scans (IntegrationTestLazyCfLoading) that could be generalized/forked for testing this scenario. E.g. with many store files. In HBASE-5416 I found some issue using that test.

        Show
        Sergey Shelukhin added a comment - Also, there's an integration test for scans (IntegrationTestLazyCfLoading) that could be generalized/forked for testing this scenario. E.g. with many store files. In HBASE-5416 I found some issue using that test.
        Hide
        Sergey Shelukhin added a comment -
        +    try {
        +      latch.await();
        ...
        +    for (KeyValueScanner scanner : scanners) {
        +      if (!(scanner instanceof StoreFileScanner)) {
        +        scanner.seek(keyValue);
        

        Can we do the sync calls before waiting for the latch for even more parallelism?

        Show
        Sergey Shelukhin added a comment - + try { + latch.await(); ... + for (KeyValueScanner scanner : scanners) { + if (!(scanner instanceof StoreFileScanner)) { + scanner.seek(keyValue); Can we do the sync calls before waiting for the latch for even more parallelism?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566762/HBASE-7495-v3.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
        org.apache.hadoop.hbase.regionserver.TestHRegion
        org.apache.hadoop.hbase.TestAcidGuarantees

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566762/HBASE-7495-v3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.TestAcidGuarantees Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4210//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566733/HBASE-7495-v2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestStoreScanner
        org.apache.hadoop.hbase.regionserver.TestHRegion
        org.apache.hadoop.hbase.TestRegionRebalancing
        org.apache.hadoop.hbase.regionserver.wal.TestHLog
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566733/HBASE-7495-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestStoreScanner org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.regionserver.wal.TestHLog org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4209//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        The latest patch fixed a MVCC related problem

        Show
        Liang Xie added a comment - The latest patch fixed a MVCC related problem
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566294/HBASE-7495.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestHRegion
        org.apache.hadoop.hbase.regionserver.TestAtomicOperation
        org.apache.hadoop.hbase.TestAcidGuarantees
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566294/HBASE-7495.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.TestAcidGuarantees org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4157//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        Really comfused to me, why does it break ACID testcase...

        Show
        Liang Xie added a comment - Really comfused to me, why does it break ACID testcase...
        Hide
        Liang Xie added a comment -

        IMHO, the order is maintained by the following:

        heap = new KeyValueHeap(scanners, store.comparator);
        
        Show
        Liang Xie added a comment - IMHO, the order is maintained by the following: heap = new KeyValueHeap(scanners, store.comparator);
        Hide
        ramkrishna.s.vasudevan added a comment -

        Liang Xie
        How will the ordering be maintained. Do we need to ensure the ordering of the kvs? Just asking?

        Show
        ramkrishna.s.vasudevan added a comment - Liang Xie How will the ordering be maintained. Do we need to ensure the ordering of the kvs? Just asking?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566118/HBASE-7495.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
        org.apache.hadoop.hbase.regionserver.TestAtomicOperation
        org.apache.hadoop.hbase.regionserver.TestHRegion
        org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
        org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
        org.apache.hadoop.hbase.TestAcidGuarantees
        org.apache.hadoop.hbase.TestLocalHBaseCluster

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566118/HBASE-7495.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.regionserver.wal.TestLogRolling org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction org.apache.hadoop.hbase.TestAcidGuarantees org.apache.hadoop.hbase.TestLocalHBaseCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4145//console This message is automatically generated.
        Hide
        Lars Hofhansl added a comment -

        I can see that for Gets it would be an improvement.

        Show
        Lars Hofhansl added a comment - I can see that for Gets it would be an improvement.
        Hide
        Liang Xie added a comment -

        Lars Hofhansl, for "get" operation, the code should go to "seek", not "requestSeek", please correct me if anything wrong
        chunhui shen, IMHO, it's easier to see the improment if the test:1)larger test dataset and smaller cache size 2)more numberOfStorefiles/disks 3)not too heavy IO-bound(e.g. 100% io util)

        the patch is broken some test cases, i'll look into it later

        Show
        Liang Xie added a comment - Lars Hofhansl , for "get" operation, the code should go to "seek", not "requestSeek", please correct me if anything wrong chunhui shen , IMHO, it's easier to see the improment if the test:1)larger test dataset and smaller cache size 2)more numberOfStorefiles/disks 3)not too heavy IO-bound(e.g. 100% io util) the patch is broken some test cases, i'll look into it later
        Hide
        Liang Xie added a comment -

        I just did a apple-to-apple comparison this morning, it shows the parallel seek reduces latency in special scenario.
        Attached is a prelim patch just for refer.

        My test env : 10 dn/rs each with 12*2T SATA, "hfile.block.cache.size=0", hbase0.94.3, cdh4.1.1
        My test data :
        recordcount=1000000000
        fieldcount=3
        fieldlength=200

        hbase(main):002:0> describe 'YCSBTest'
        DESCRIPTION ENABLED
        {NAME => 'YCSBTest', SPLIT_POLICY => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy', FAMILIES => [

        {NAME => 'te true st', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '1', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VER SIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'tru e', BLOCKCACHE => 'true'}

        ]}

        $./hdfs dfs -du -s -h hdfs://lgxl-xieliang/
        726.8g hdfs://lgxl-xieliang/

        100 regions in total, and most of numberOfStorefiles in those regions are [0,5]

        My test cmd: bin/ycsb run hbase -P ./workloads/kaka -threads 1 -p columnfamily=test -p table=YCSBTest -s > log/run.log 2>&1 &

        I restarted the whole hbase/hdfs cluster and clear OS cache(echo 1 > /proc/sys/vm/drop_caches) before each run.

        Serial seek result:

        [OVERALL], RunTime(ms), 300027.0
        [OVERALL], Throughput(ops/sec), 20.09819116279535
        [READ], Operations, 6030
        [READ], AverageLatency(us), 49739.97446102819
        [READ], MinLatency(us), 2768
        [READ], MaxLatency(us), 782892
        [READ], 50thPercentileLatency(ms), 45
        [READ], 95thPercentileLatency(ms), 90
        [READ], 99thPercentileLatency(ms), 124
        [READ], Return=0, 6030

        Parallel seek result:

        [OVERALL], RunTime(ms), 300016.0
        [OVERALL], Throughput(ops/sec), 39.584555490373845
        [READ], Operations, 11876
        [READ], AverageLatency(us), 25249.878410239136
        [READ], MinLatency(us), 3084
        [READ], MaxLatency(us), 753547
        [READ], 50thPercentileLatency(ms), 22
        [READ], 95thPercentileLatency(ms), 43
        [READ], 99thPercentileLatency(ms), 67
        [READ], Return=0, 11876

        Show
        Liang Xie added a comment - I just did a apple-to-apple comparison this morning, it shows the parallel seek reduces latency in special scenario. Attached is a prelim patch just for refer. My test env : 10 dn/rs each with 12*2T SATA, "hfile.block.cache.size=0", hbase0.94.3, cdh4.1.1 My test data : recordcount=1000000000 fieldcount=3 fieldlength=200 hbase(main):002:0> describe 'YCSBTest' DESCRIPTION ENABLED {NAME => 'YCSBTest', SPLIT_POLICY => 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy', FAMILIES => [ {NAME => 'te true st', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '1', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VER SIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'tru e', BLOCKCACHE => 'true'} ]} $./hdfs dfs -du -s -h hdfs://lgxl-xieliang/ 726.8g hdfs://lgxl-xieliang/ 100 regions in total, and most of numberOfStorefiles in those regions are [0,5] My test cmd: bin/ycsb run hbase -P ./workloads/kaka -threads 1 -p columnfamily=test -p table=YCSBTest -s > log/run.log 2>&1 & I restarted the whole hbase/hdfs cluster and clear OS cache(echo 1 > /proc/sys/vm/drop_caches) before each run. Serial seek result: [OVERALL] , RunTime(ms), 300027.0 [OVERALL] , Throughput(ops/sec), 20.09819116279535 [READ] , Operations, 6030 [READ] , AverageLatency(us), 49739.97446102819 [READ] , MinLatency(us), 2768 [READ] , MaxLatency(us), 782892 [READ] , 50thPercentileLatency(ms), 45 [READ] , 95thPercentileLatency(ms), 90 [READ] , 99thPercentileLatency(ms), 124 [READ] , Return=0, 6030 Parallel seek result: [OVERALL] , RunTime(ms), 300016.0 [OVERALL] , Throughput(ops/sec), 39.584555490373845 [READ] , Operations, 11876 [READ] , AverageLatency(us), 25249.878410239136 [READ] , MinLatency(us), 3084 [READ] , MaxLatency(us), 753547 [READ] , 50thPercentileLatency(ms), 22 [READ] , 95thPercentileLatency(ms), 43 [READ] , 99thPercentileLatency(ms), 67 [READ] , Return=0, 11876
        Hide
        chunhui shen added a comment -

        Liang Xie
        We have do the parallel scanner seek testing long ago, but seems no improvement.

        Waiting for your new result.

        Show
        chunhui shen added a comment - Liang Xie We have do the parallel scanner seek testing long ago, but seems no improvement. Waiting for your new result.
        Hide
        Liang Xie added a comment -

        Oops, seems i have some misunderstanding on this code snippet, emm, i have to debug it per your comments

        Show
        Liang Xie added a comment - Oops, seems i have some misunderstanding on this code snippet, emm, i have to debug it per your comments
        Hide
        Lars Hofhansl added a comment -

        In that case it might help, but note that this seek is only done when the scanner is opened (which happens once per region), so even there the saving will not be significant (my guess).

        Do you have any performance numbers that concern you?

        Show
        Lars Hofhansl added a comment - In that case it might help, but note that this seek is only done when the scanner is opened (which happens once per region), so even there the saving will not be significant (my guess). Do you have any performance numbers that concern you?
        Hide
        Liang Xie added a comment -

        How about "scanner.seek" Lars Hofhansl, we seek memstore and storefiles one by one currently.

        Show
        Liang Xie added a comment - How about "scanner.seek" Lars Hofhansl , we seek memstore and storefiles one by one currently.
        Hide
        Lars Hofhansl added a comment -

        requestSeek(matcher.getStartKey(), false, true) is hardly doing anything, I do not think we'll see any measurable improvement from this.

        What generally might help is to scan families in parallel (rather than serially the way we do that now), but that is very tricky.

        Show
        Lars Hofhansl added a comment - requestSeek(matcher.getStartKey(), false, true) is hardly doing anything, I do not think we'll see any measurable improvement from this. What generally might help is to scan families in parallel (rather than serially the way we do that now), but that is very tricky.
        Hide
        Liang Xie added a comment -

        The parallel manner does not translate into an overall throughput improvement,IMHO

        Show
        Liang Xie added a comment - The parallel manner does not translate into an overall throughput improvement,IMHO

          People

          • Assignee:
            Liang Xie
            Reporter:
            Liang Xie
          • Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development