HBase
  1. HBase
  2. HBASE-11811

Use binary search for seeking into a block

    Details

    • Type: Brainstorming Brainstorming
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently upon every seek (including Gets) we need to linearly look through the block from the beginning until we find the Cell we are looking for.

      It should be possible to build a simple cache of offsets of Cells for each block as it is loaded and then use binary search to find the Cell in question.

      1. 11811-wip-v2.txt
        16 kB
        Lars Hofhansl
      2. 11811-wip-v4.txt
        19 kB
        Lars Hofhansl
      3. block_index-v2.txt
        9 kB
        Lars Hofhansl

        Issue Links

          Activity

          Hide
          Lars Hofhansl added a comment - - edited

          Here's sample patch that reduces the time for 1m gets from 40s to 8.5s and there is probably more room for optimization.
          The data was simply generated by HBaseTestingUtility.loadTable(...) so the KVs are small.

          Some points:

          1. the utility of this decreases as Cells get larger and only a few of them fit into a block
          2. the index is not persisted, so when blocks are evicted and later reloaded the index needs to be build up again
          3. not happy currently about the point where the index is built, as that needs to synchronize on the block (but only when the block actually had to be loaded)
          Show
          Lars Hofhansl added a comment - - edited Here's sample patch that reduces the time for 1m gets from 40s to 8.5s and there is probably more room for optimization. The data was simply generated by HBaseTestingUtility.loadTable(...) so the KVs are small. Some points: the utility of this decreases as Cells get larger and only a few of them fit into a block the index is not persisted, so when blocks are evicted and later reloaded the index needs to be build up again not happy currently about the point where the index is built, as that needs to synchronize on the block (but only when the block actually had to be loaded)
          Hide
          Lars Hofhansl added a comment -

          I also just renamed the old blockSeek methods to blockSeekOld, obviously those would be removed.

          Looking through the code I noticed a lot of duplication betweek HFileReaderV2 and HFileReaderV3, this can probably factored a bit nicer generally.

          Show
          Lars Hofhansl added a comment - I also just renamed the old blockSeek methods to blockSeekOld, obviously those would be removed. Looking through the code I noticed a lot of duplication betweek HFileReaderV2 and HFileReaderV3, this can probably factored a bit nicer generally.
          Hide
          Lars Hofhansl added a comment -

          And lastly, this obviously does not work when any block encoding is used as there is no offset for a Cell, so the code is only for ScannerVx and not EncodedScannerVx.

          Show
          Lars Hofhansl added a comment - And lastly, this obviously does not work when any block encoding is used as there is no offset for a Cell, so the code is only for ScannerVx and not EncodedScannerVx.
          Hide
          Lars Hofhansl added a comment -

          When I do larger KV (values of 90 bytes) the old code path takes 19.5s (there are fewer keys to compare compared to the value size). With the patch it takes 8.3s.

          Show
          Lars Hofhansl added a comment - When I do larger KV (values of 90 bytes) the old code path takes 19.5s (there are fewer keys to compare compared to the value size). With the patch it takes 8.3s.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663838/block_index-v2.txt
          against trunk revision .
          ATTACHMENT ID: 12663838

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 6 warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.io.hfile.TestHFileBlock

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663838/block_index-v2.txt against trunk revision . ATTACHMENT ID: 12663838 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 6 warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileBlock Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10548//console This message is automatically generated.
          Hide
          Lars Hofhansl added a comment -

          Fixes TestHFileBlock. Possible improvements:

          • build index as we go to avoid building the complete index when only a single Cell is read from a block
          • persist the index. At write time we can pretty much get the offsets without extra cost
          • make this configurable...? We're adding 4 bytes per Cell of heap usage.
          Show
          Lars Hofhansl added a comment - Fixes TestHFileBlock. Possible improvements: build index as we go to avoid building the complete index when only a single Cell is read from a block persist the index. At write time we can pretty much get the offsets without extra cost make this configurable...? We're adding 4 bytes per Cell of heap usage.
          Hide
          Lars Hofhansl added a comment -

          This version gets rid of the unused code and also makes block seeking identical between HFileReaderV2 and HFileReaderV3.

          Show
          Lars Hofhansl added a comment - This version gets rid of the unused code and also makes block seeking identical between HFileReaderV2 and HFileReaderV3.
          Hide
          Andrew Purtell added a comment -

          Interesting. We should take this for a spin on a small cluster with YCSB.

          also makes block seeking identical between HFileReaderV2 and HFileReaderV3.

          We should clean that up in any case.

          Show
          Andrew Purtell added a comment - Interesting. We should take this for a spin on a small cluster with YCSB. also makes block seeking identical between HFileReaderV2 and HFileReaderV3. We should clean that up in any case.
          Hide
          Lars Hofhansl added a comment -

          Yeah, PE or YCSB is the next step. I expect with the current patch scanning would be slower as we would build up the index every time we switch to a new block. Might need a flag to updateCurrentBlock to indicate whether we want to load the index or so.
          I think the ondemand build of the index would be best (probably need to bring back the current seek code again in that case).

          I suppose I could make a separate patch just removing code duplication between ReaderV2 and V3.

          Show
          Lars Hofhansl added a comment - Yeah, PE or YCSB is the next step. I expect with the current patch scanning would be slower as we would build up the index every time we switch to a new block. Might need a flag to updateCurrentBlock to indicate whether we want to load the index or so. I think the ondemand build of the index would be best (probably need to bring back the current seek code again in that case). I suppose I could make a separate patch just removing code duplication between ReaderV2 and V3.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663969/11811-wip-v2.txt
          against trunk revision .
          ATTACHMENT ID: 12663969

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 7 warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663969/11811-wip-v2.txt against trunk revision . ATTACHMENT ID: 12663969 +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 7 warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10553//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663968/11811-wip.txt
          against trunk revision .
          ATTACHMENT ID: 12663968

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 6 warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + blockBuffer.position(blockBuffer.position() + currKeyLen + currValueLen + currMemstoreTSLen + KEY_VALUE_LEN_SIZE);
          + byte[] v = Bytes.toBytes("Some string that is longer than just three characters and makes this a more real scenario...");

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.client.TestFromClientSide
          org.apache.hadoop.hbase.TestRegionRebalancing
          org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663968/11811-wip.txt against trunk revision . ATTACHMENT ID: 12663968 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 6 warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + blockBuffer.position(blockBuffer.position() + currKeyLen + currValueLen + currMemstoreTSLen + KEY_VALUE_LEN_SIZE); + byte[] v = Bytes.toBytes("Some string that is longer than just three characters and makes this a more real scenario..."); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10552//console This message is automatically generated.
          Hide
          Andrew Purtell added a comment -

          I suppose I could make a separate patch just removing code duplication between ReaderV2 and V3

          +1

          Show
          Andrew Purtell added a comment - I suppose I could make a separate patch just removing code duplication between ReaderV2 and V3 +1
          Hide
          Lars Hofhansl added a comment -

          In order to work fine for scanning:

          1. avoid this unless the caller wants to seek into the block. A simple next() into the next block should not cause a prefetch of all the locations of all Cells
          2. also avoid this when the block is not cached in the first place
          Show
          Lars Hofhansl added a comment - In order to work fine for scanning: avoid this unless the caller wants to seek into the block. A simple next() into the next block should not cause a prefetch of all the locations of all Cells also avoid this when the block is not cached in the first place
          Hide
          Lars Hofhansl added a comment -

          Found HBASE-9679, which also discussed the issue. Completely forgot about that.

          Show
          Lars Hofhansl added a comment - Found HBASE-9679 , which also discussed the issue. Completely forgot about that.
          Hide
          Lars Hofhansl added a comment -

          Slightly updated version.

          In all my local testing I see great improvement for Gets and no measurable penalty for Scans.

          In the worst case (if each block only receives a single Get request) we can expect that on average we now look through twice as many KV. If the block is indeed loaded from HDFS each time, this is negligible.

          I'll try to test with a real cluster and PE soon.

          Show
          Lars Hofhansl added a comment - Slightly updated version. In all my local testing I see great improvement for Gets and no measurable penalty for Scans. In the worst case (if each block only receives a single Get request) we can expect that on average we now look through twice as many KV. If the block is indeed loaded from HDFS each time, this is negligible. I'll try to test with a real cluster and PE soon.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665272/11811-wip-v4.txt
          against trunk revision .
          ATTACHMENT ID: 12665272

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          -1 lineLengths. The patch introduces the following lines longer than 100:
          + blockBuffer.position(blockBuffer.position() + currKeyLen + currValueLen + currMemstoreTSLen + KEY_VALUE_LEN_SIZE);
          + //byte[] v = Bytes.toBytes("Some string that is longer than just three characters and makes this a more real scenario...");
          + //byte[] v = Bytes.toBytes("Some string that is longer than just three characters and makes this a more real scenario...");

          +1 site. The mvn site goal succeeds with this patch.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.TestClassFinder

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665272/11811-wip-v4.txt against trunk revision . ATTACHMENT ID: 12665272 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 lineLengths . The patch introduces the following lines longer than 100: + blockBuffer.position(blockBuffer.position() + currKeyLen + currValueLen + currMemstoreTSLen + KEY_VALUE_LEN_SIZE); + //byte[] v = Bytes.toBytes("Some string that is longer than just three characters and makes this a more real scenario..."); + //byte[] v = Bytes.toBytes("Some string that is longer than just three characters and makes this a more real scenario..."); +1 site . The mvn site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.TestClassFinder Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10636//console This message is automatically generated.
          Hide
          stack added a comment -

          Patch looks good. Would help case where repeated, random seeks in a small set of hot blocks. Won't help scans.

          We should write the index into the block at write time rather than calculate when reading?

          Show
          stack added a comment - Patch looks good. Would help case where repeated, random seeks in a small set of hot blocks. Won't help scans. We should write the index into the block at write time rather than calculate when reading?

            People

            • Assignee:
              Unassigned
              Reporter:
              Lars Hofhansl
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:

                Development