Hadoop Common
  1. Hadoop Common
  2. HADOOP-6546

BloomMapFile can return false negatives

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 1.1.0, 0.21.0
    • Component/s: io
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      BloomMapFile can return false negatives when using keys of varying sizes. If the amount of data written by the write() method of your key class differs between instance of your key, your BloomMapFile may return false negatives.

      1. HADOOP-6546.patch
        3 kB
        Clark Jefcoat

        Activity

        Clark Jefcoat created issue -
        Hide
        Clark Jefcoat added a comment -

        The issue is with the call
        {{
        bloomKey.set(buf.getData(), 1.0);
        }}
        which appears twice in the BloomMapFile source. The buf variable is a DataOutputBuffer. The documentation for DataOutputBuffer clearly states that getData() is only valid to getLength(). But bloomKey is an o.a.h.util.bloom.Key which expects the entire array that it is getting to be valid.

        Show
        Clark Jefcoat added a comment - The issue is with the call {{ bloomKey.set(buf.getData(), 1.0); }} which appears twice in the BloomMapFile source. The buf variable is a DataOutputBuffer. The documentation for DataOutputBuffer clearly states that getData() is only valid to getLength(). But bloomKey is an o.a.h.util.bloom.Key which expects the entire array that it is getting to be valid.
        Hide
        Clark Jefcoat added a comment -

        Simple tests to demonstrate the problem and a proposed solution are attached.

        Show
        Clark Jefcoat added a comment - Simple tests to demonstrate the problem and a proposed solution are attached.
        Clark Jefcoat made changes -
        Field Original Value New Value
        Attachment HADOOP-6546.patch [ 12435190 ]
        Hide
        Todd Lipcon added a comment -

        Marking patch available so the Hudson QA bot tests this.

        Show
        Todd Lipcon added a comment - Marking patch available so the Hudson QA bot tests this.
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12435190/HADOOP-6546.patch
        against trunk revision 907549.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12435190/HADOOP-6546.patch against trunk revision 907549. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/345/console This message is automatically generated.
        Hide
        Tom White added a comment -

        +1 I've just committed this. Thanks Clark!

        Show
        Tom White added a comment - +1 I've just committed this. Thanks Clark!
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Assignee Clark Jefcoat [ cjjefcoat ]
        Fix Version/s 0.22.0 [ 12314296 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #203 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/203/)
        . BloomMapFile can return false negatives. Contributed by Clark Jefcoat.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #203 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/203/ ) . BloomMapFile can return false negatives. Contributed by Clark Jefcoat.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #282 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/282/)
        . BloomMapFile can return false negatives. Contributed by Clark Jefcoat.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #282 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/282/ ) . BloomMapFile can return false negatives. Contributed by Clark Jefcoat.
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Fix Version/s 0.22.0 [ 12314296 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Suresh Srinivas made changes -
        Target Version/s 1.1.0 [ 12316501 ]
        Hide
        Suresh Srinivas added a comment -

        I committed this patch to branch-1. It should be available in release 1.1.

        Show
        Suresh Srinivas added a comment - I committed this patch to branch-1. It should be available in release 1.1.
        Suresh Srinivas made changes -
        Fix Version/s 1.1.0 [ 12316501 ]

          People

          • Assignee:
            Clark Jefcoat
            Reporter:
            Clark Jefcoat
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development