HBase
  1. HBase
  2. HBASE-6359

KeyValue may return incorrect values after readFields()

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When the same KeyValue object is used multiple times for deserialization using readFields, some methods may return incorrect values. Here is a sequence of operations that will reproduce the problem:

      1. A KeyValue is created whose key has length 10. The private field keyLength is initialized to 0.
      2. KeyValue.getKeyLength() is called. This reads the key length 10 from the backing array and caches it in keyLength.
      3. KeyValue.readFields() is called to deserialize a new value. The keyLength field is not cleared and keeps its value of 10, even though this value is probably incorrect.
      4. If getKeyLength() is called, the value 10 will be returned.

      For example, in a reducer with Iterable<KeyValue>, all values after the first one from the iterable are likely to return incorrect values from getKeyLength().

      The solution is to clear all memoized values in KeyValue.readFields(). I'll write a patch for this soon.

        Activity

        Hide
        Dave Revell added a comment -

        Attached HBASE-6359-trunk-v1.diff

        Show
        Dave Revell added a comment - Attached HBASE-6359 -trunk-v1.diff
        Hide
        Ted Yu added a comment -

        Patch looks good to me.

        Show
        Ted Yu added a comment - Patch looks good to me.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538974/HBASE-6359-trunk-v1.diff
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).

        -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538974/HBASE-6359-trunk-v1.diff against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2491//console This message is automatically generated.
        Hide
        Ted Yu added a comment -

        Will integrate tomorrow if there is no objection.

        Show
        Ted Yu added a comment - Will integrate tomorrow if there is no objection.
        Hide
        Lars Hofhansl added a comment -

        +1

        (Personally I'd rather get rid of these optimizations for rowCache, timestampCache, and keyLength. But that's for another jira)

        Show
        Lars Hofhansl added a comment - +1 (Personally I'd rather get rid of these optimizations for rowCache, timestampCache, and keyLength. But that's for another jira)
        Hide
        Ted Yu added a comment -

        Integrated to 0.94 and trunk.

        Thanks for the patch, Dave.

        Thanks for the review, Lars.

        Show
        Ted Yu added a comment - Integrated to 0.94 and trunk. Thanks for the patch, Dave. Thanks for the review, Lars.
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #3194 (See https://builds.apache.org/job/HBase-TRUNK/3194/)
        HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368964)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #3194 (See https://builds.apache.org/job/HBase-TRUNK/3194/ ) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368964) Result = SUCCESS tedyu : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94 #381 (See https://builds.apache.org/job/HBase-0.94/381/)
        HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Show
        Hudson added a comment - Integrated in HBase-0.94 #381 (See https://builds.apache.org/job/HBase-0.94/381/ ) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #120 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/120/)
        HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368964)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #120 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/120/ ) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368964) Result = FAILURE tedyu : Files : /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/)
        HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967)

        Result = FAILURE
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security-on-Hadoop-23 #6 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/6/ ) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Hide
        Hudson added a comment -

        Integrated in HBase-0.94-security #47 (See https://builds.apache.org/job/HBase-0.94-security/47/)
        HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967)

        Result = SUCCESS
        tedyu :
        Files :

        • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Show
        Hudson added a comment - Integrated in HBase-0.94-security #47 (See https://builds.apache.org/job/HBase-0.94-security/47/ ) HBASE-6359 KeyValue may return incorrect values after readFields() (Dave Revell) (Revision 1368967) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java
        Hide
        stack added a comment -

        Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)

        Show
        stack added a comment - Fix up after bulk move overwrote some 0.94.2 fix versions w/ 0.95.0 (Noticed by Lars Hofhansl)

          People

          • Assignee:
            Dave Revell
            Reporter:
            Dave Revell
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development