Hadoop Common
  1. Hadoop Common
  2. HADOOP-5657

Validate data passed through TestReduceFetch

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.

      1. 5657-0.patch
        9 kB
        Chris Douglas
      2. 5657-1.patch
        8 kB
        Chris Douglas
      3. 5657-2.patch
        9 kB
        Chris Douglas

        Activity

        Hide
        Chris Douglas added a comment -

        The following are now validated in the reduce:

        • Each map produces one record for each of 4096 small keys
        • Includes unique large records, each straddled by a pair of small records from another map (to detect corruption from the merge)
        • Changes some parameters for testReduceFromDisk to make intermediate merges with in-memory data occur occasionally
        Show
        Chris Douglas added a comment - The following are now validated in the reduce: Each map produces one record for each of 4096 small keys Includes unique large records, each straddled by a pair of small records from another map (to detect corruption from the merge) Changes some parameters for testReduceFromDisk to make intermediate merges with in-memory data occur occasionally
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12405214/5657-0.patch
        against trunk revision 764287.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405214/5657-0.patch against trunk revision 764287. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        Emits only two additional records per map; less expensive key updates. Detects known-bad case when merging a combination of in-memory and on-disk segments.

        Show
        Chris Douglas added a comment - Emits only two additional records per map; less expensive key updates. Detects known-bad case when merging a combination of in-memory and on-disk segments.
        Hide
        Devaraj Das added a comment -

        +1

        Show
        Devaraj Das added a comment - +1
        Hide
        Devaraj Das added a comment -

        It'd help code readability if some comments are added on why two values are emitted per map and the logic the testcase is employing for validation (verbose comments for the arithmetic is what I mean smile).

        Show
        Devaraj Das added a comment - It'd help code readability if some comments are added on why two values are emitted per map and the logic the testcase is employing for validation (verbose comments for the arithmetic is what I mean smile ).
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12405486/5657-1.patch
        against trunk revision 765815.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405486/5657-1.patch against trunk revision 765815. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        Adds some extra comments; no functional changes

        Show
        Chris Douglas added a comment - Adds some extra comments; no functional changes
        Hide
        Chris Douglas added a comment -

        I committed this.

        Show
        Chris Douglas added a comment - I committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/)
        . Validate data in TestReduceFetch to improve merge test coverage.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/ ) . Validate data in TestReduceFetch to improve merge test coverage.

          People

          • Assignee:
            Chris Douglas
            Reporter:
            Chris Douglas
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development