Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.0, 0.96.1
    • Fix Version/s: 0.98.0
    • Component/s: regionserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      make generating faked key algo more aggressive

      1. HBASE-9518-v2.txt
        7 kB
        Liang Xie
      2. HBASE-9518.txt
        5 kB
        Liang Xie

        Activity

        Liang Xie created issue -
        Liang Xie made changes -
        Field Original Value New Value
        Description make generating faked key algo more aggressive
        Liang Xie made changes -
        Attachment HBASE-9518.txt [ 12602750 ]
        Liang Xie made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Liang Xie [ xieliang007 ]
        Hide
        Liang Xie added a comment -

        TestCacheOnWrite case is updated due to the index size is smaller after applied the more aggressive change.

        Show
        Liang Xie added a comment - TestCacheOnWrite case is updated due to the index size is smaller after applied the more aggressive change.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12602750/HBASE-9518.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        -1 site. The patch appears to cause mvn site goal to fail.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.TestHalfStoreFileReader

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12602750/HBASE-9518.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 6 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. -1 core tests . The patch failed these unit tests: org.apache.hadoop.hbase.io.TestHalfStoreFileReader Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7180//console This message is automatically generated.
        Hide
        Liang Xie added a comment -

        seems it depends on a little other change, i just port the related from our internal codebase, let's try another QA run for v2

        Show
        Liang Xie added a comment - seems it depends on a little other change, i just port the related from our internal codebase, let's try another QA run for v2
        Liang Xie made changes -
        Attachment HBASE-9518-v2.txt [ 12602773 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12602773/HBASE-9518-v2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 lineLengths. The patch does not introduce lines longer than 100

        +1 site. The mvn site goal succeeds with this patch.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12602773/HBASE-9518-v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 9 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7185//console This message is automatically generated.
        Hide
        Ted Yu added a comment -
        +      System.out.println("midkey: " + midKV + " or: " + Bytes.toStringBinary(midkey));
        +      System.out.println("beforeMidKey: " + beforeMidKey);
        

        Can you replace the above with LOG ?

        Show
        Ted Yu added a comment - + System .out.println( "midkey: " + midKV + " or: " + Bytes.toStringBinary(midkey)); + System .out.println( "beforeMidKey: " + beforeMidKey); Can you replace the above with LOG ?
        Hide
        stack added a comment -

        make generating faked key algo more aggressive

        What does this mean Liang Xie? Thanks.

        Show
        stack added a comment - make generating faked key algo more aggressive What does this mean Liang Xie ? Thanks.
        Hide
        Liang Xie added a comment -

        Hi stack you can see the new TestKeyValue case:
        if the last kv of previous block and the first kv of current block have same postfix and just 1 offset diff, e.g. 100abcdefg and 101abcdefg,
        before 9518, the getShortMidpointKey() will fallback to the default right kv, say 101abcdefg.
        after 9518, it'll return "101", a shorter faked value, still reasonable, right?
        And i found this corner case existing in current hbase test cases as well, so i'd like to let it go into community codebase also.

        Show
        Liang Xie added a comment - Hi stack you can see the new TestKeyValue case: if the last kv of previous block and the first kv of current block have same postfix and just 1 offset diff, e.g. 100abcdefg and 101abcdefg, before 9518, the getShortMidpointKey() will fallback to the default right kv, say 101abcdefg. after 9518, it'll return "101", a shorter faked value, still reasonable, right? And i found this corner case existing in current hbase test cases as well, so i'd like to let it go into community codebase also.
        Hide
        Liang Xie added a comment -

        any more comments? thanks

        Show
        Liang Xie added a comment - any more comments? thanks
        Hide
        stack added a comment -

        lgtm Makes sense (I see you left an 'opportunity for improvement' comment that this takes care of). Your explanation above should become the release note. Anyone else want to check it out? I'll commit tomorrow otherwise to trunk.

        Show
        stack added a comment - lgtm Makes sense (I see you left an 'opportunity for improvement' comment that this takes care of). Your explanation above should become the release note. Anyone else want to check it out? I'll commit tomorrow otherwise to trunk.
        Hide
        Jean-Daniel Cryans added a comment -

        +1

        When you have some time Liang Xie, it'd be nice if you added the faked keys to this documentation http://hbase.apache.org/book.html#hfilev2

        Show
        Jean-Daniel Cryans added a comment - +1 When you have some time Liang Xie , it'd be nice if you added the faked keys to this documentation http://hbase.apache.org/book.html#hfilev2
        Hide
        Liang Xie added a comment -

        Jean-Daniel Cryans, thanks for review, i just filed HBASE-9583 to add the document.

        Show
        Liang Xie added a comment - Jean-Daniel Cryans , thanks for review, i just filed HBASE-9583 to add the document.
        Hide
        stack added a comment -

        Committed to trunk. Thank you Liang Xie

        Show
        stack added a comment - Committed to trunk. Thank you Liang Xie
        stack made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.98.0 [ 12323143 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK #4535 (See https://builds.apache.org/job/HBase-TRUNK/4535/)
        HBASE-9518 getFakedKey() improvement (stack: rev 1524896)

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        • /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4535 (See https://builds.apache.org/job/HBase-TRUNK/4535/ ) HBASE-9518 getFakedKey() improvement (stack: rev 1524896) /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #743 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/743/)
        HBASE-9518 getFakedKey() improvement (stack: rev 1524896)

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        • /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java
        • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java
        • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
        Show
        Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #743 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/743/ ) HBASE-9518 getFakedKey() improvement (stack: rev 1524896) /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/TestKeyValue.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        44m 38s 1 Liang Xie 12/Sep/13 09:05
        Patch Available Patch Available Resolved Resolved
        7d 19h 42m 1 stack 20/Sep/13 04:47

          People

          • Assignee:
            Liang Xie
            Reporter:
            Liang Xie
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development