HBase
  1. HBase
  2. HBASE-6032

Port HFileBlockIndex improvement from HBASE-5987

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.3, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Excerpt from HBASE-5987:
      First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is "smaller" than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. Indexing on the start key has some defects here) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is "bigger", then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup.

      This JIRA is to port the fix to HBase trunk, etc.

      1. 6032.094.txt
        34 kB
        stack
      2. 6032-ports-5987.txt
        32 kB
        Ted Yu
      3. 6032-ports-5987-v2.txt
        32 kB
        Ted Yu
      4. 6032v3.txt
        34 kB
        stack

        Issue Links

          Activity

          Hide
          stack added a comment -

          Vladimir Rodionov asks if this forward port actually works as expected over in HBASE-9769? Do we know? Was the benefit comfirmed on commit?

          Show
          stack added a comment - Vladimir Rodionov asks if this forward port actually works as expected over in HBASE-9769 ? Do we know? Was the benefit comfirmed on commit?
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/)
          HBASE-6032 Addendum, forgot to add two files (Revision 1399521)
          HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java

          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/ ) HBASE-6032 Addendum, forgot to add two files (Revision 1399521) HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
          Hide
          Lars Hofhansl added a comment -

          Ran queueFailover manually. Passes locally.

          Show
          Lars Hofhansl added a comment - Ran queueFailover manually. Passes locally.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #540 (See https://builds.apache.org/job/HBase-0.94/540/)
          HBASE-6032 Addendum, forgot to add two files (Revision 1399521)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #540 (See https://builds.apache.org/job/HBase-0.94/540/ ) HBASE-6032 Addendum, forgot to add two files (Revision 1399521) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
          Hide
          Lars Hofhansl added a comment -

          Whoops. Forgot to "svn add" the two new files. Done through an addendum.

          Show
          Lars Hofhansl added a comment - Whoops. Forgot to "svn add" the two new files. Done through an addendum.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #539 (See https://builds.apache.org/job/HBase-0.94/539/)
          HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513)

          Result = FAILURE
          larsh :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #539 (See https://builds.apache.org/job/HBase-0.94/539/ ) HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513) Result = FAILURE larsh : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
          Hide
          Lars Hofhansl added a comment -

          Committed to 0.94 as well.

          Show
          Lars Hofhansl added a comment - Committed to 0.94 as well.
          Hide
          Lars Hofhansl added a comment -

          I ran the relevant tests. And also used with some other test cases I have. (The set was with a local HDFS, so I didn't observe any performance benefits)

          Show
          Lars Hofhansl added a comment - I ran the relevant tests. And also used with some other test cases I have. (The set was with a local HDFS, so I didn't observe any performance benefits)
          Hide
          Lars Hofhansl added a comment -

          Awesome. That was quick. Thanks Stack!
          I'll test out a bit today and then commit if all is good.

          Show
          Lars Hofhansl added a comment - Awesome. That was quick. Thanks Stack! I'll test out a bit today and then commit if all is good.
          Hide
          stack added a comment -

          I've not run the tests but +1 on commit if all tests run (nice tests included w/ this patch)

          Show
          stack added a comment - I've not run the tests but +1 on commit if all tests run (nice tests included w/ this patch)
          Hide
          stack added a comment -

          Version of patch that will apply to 0.94

          Show
          stack added a comment - Version of patch that will apply to 0.94
          Hide
          Anoop Sam John added a comment -

          +1 for having this for 0.94 version..
          In fact I was trying to make a port and test and as per that raise a new issue for porting..

          Show
          Anoop Sam John added a comment - +1 for having this for 0.94 version.. In fact I was trying to make a port and test and as per that raise a new issue for porting..
          Hide
          ramkrishna.s.vasudevan added a comment -

          We need this for 0.94 i think as per the changes in HBASE-6577.

          Show
          ramkrishna.s.vasudevan added a comment - We need this for 0.94 i think as per the changes in HBASE-6577 .
          Hide
          Lars Hofhansl added a comment -

          How come we missed this for 0.94?
          This looks like an important performance improvement.

          Show
          Lars Hofhansl added a comment - How come we missed this for 0.94? This looks like an important performance improvement.
          Hide
          stack added a comment -

          This was committed a good while back.

          Show
          stack added a comment - This was committed a good while back.
          Hide
          stack added a comment -

          @Liyin Yeah. We should port them over. One day!

          Show
          stack added a comment - @Liyin Yeah. We should port them over. One day!
          Hide
          Liyin Tang added a comment -

          Thanks Ted for porting HBASE-5987 to trunk !
          Some of the static functions in TestHBaseCase are still quite useful. It allows the unit test play around in the region level.
          Maybe we shall move these useful code to HBaseTestingUtility

          Show
          Liyin Tang added a comment - Thanks Ted for porting HBASE-5987 to trunk ! Some of the static functions in TestHBaseCase are still quite useful. It allows the unit test play around in the region level. Maybe we shall move these useful code to HBaseTestingUtility
          Hide
          stack added a comment -

          Thanks for forward porting Ted.

          Weird this patch uses TestHBaseCase and does all this fixup in it – its a deprecated class – and that it makes data members static.

          Otherwise, patch is great on cursory review.

          Show
          stack added a comment - Thanks for forward porting Ted. Weird this patch uses TestHBaseCase and does all this fixup in it – its a deprecated class – and that it makes data members static. Otherwise, patch is great on cursory review.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #30 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/30/)
          HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Revision 1343413)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #30 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/30/ ) HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Revision 1343413) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2941 (See https://builds.apache.org/job/HBase-TRUNK/2941/)
          HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Revision 1343413)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2941 (See https://builds.apache.org/job/HBase-TRUNK/2941/ ) HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Revision 1343413) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HConstants.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java
          Hide
          Ted Yu added a comment -

          Integrated patch v2 to trunk.

          Show
          Ted Yu added a comment - Integrated patch v2 to trunk.
          Hide
          Ted Yu added a comment -

          I plan to integrate patch v2 tomorrow if there is no objection.

          Show
          Ted Yu added a comment - I plan to integrate patch v2 tomorrow if there is no objection.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12529917/6032-ports-5987-v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2021//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2021//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529917/6032-ports-5987-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2021//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2021//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Rebase for trunk.

          Show
          Ted Yu added a comment - Rebase for trunk.
          Hide
          Ted Yu added a comment -

          Can someone review the port please ?

          Show
          Ted Yu added a comment - Can someone review the port please ?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12529563/6032-ports-5987.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          +1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 36 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12529563/6032-ports-5987.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 36 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1992//console This message is automatically generated.

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development