Hadoop Common
  1. Hadoop Common
  2. HADOOP-5657

Validate data passed through TestReduceFetch

        Details

        • Type: Improvement Improvement
        • Status: Closed
        • Priority: Minor Minor
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.21.0
        • Component/s: test
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          While TestReduceFetch verifies the reduce semantics for reducing from in-memory segments, it does not validate the data it reads. Data corrupted during the merge will not be detected.

          1. Text File
            5657-0.patch
            9 kB
            Chris Douglas
          2. Text File
            5657-1.patch
            8 kB
            Chris Douglas
          3. Text File
            5657-2.patch
            9 kB
            Chris Douglas

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Chris Douglas added a comment -

            The following are now validated in the reduce:

            • Each map produces one record for each of 4096 small keys
            • Includes unique large records, each straddled by a pair of small records from another map (to detect corruption from the merge)
            • Changes some parameters for testReduceFromDisk to make intermediate merges with in-memory data occur occasionally
            Show
            Chris Douglas added a comment - The following are now validated in the reduce: Each map produces one record for each of 4096 small keys Includes unique large records, each straddled by a pair of small records from another map (to detect corruption from the merge) Changes some parameters for testReduceFromDisk to make intermediate merges with in-memory data occur occasionally
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12405214/5657-0.patch
            against trunk revision 764287.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            -1 core tests. The patch failed core unit tests.

            -1 contrib tests. The patch failed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405214/5657-0.patch against trunk revision 764287. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/186/console This message is automatically generated.
            Hide
            Permalink
            Chris Douglas added a comment -

            Emits only two additional records per map; less expensive key updates. Detects known-bad case when merging a combination of in-memory and on-disk segments.

            Show
            Chris Douglas added a comment - Emits only two additional records per map; less expensive key updates. Detects known-bad case when merging a combination of in-memory and on-disk segments.
            Hide
            Permalink
            Devaraj Das added a comment -

            +1

            Show
            Devaraj Das added a comment - +1
            Hide
            Permalink
            Devaraj Das added a comment -

            It'd help code readability if some comments are added on why two values are emitted per map and the logic the testcase is employing for validation (verbose comments for the arithmetic is what I mean smile).

            Show
            Devaraj Das added a comment - It'd help code readability if some comments are added on why two values are emitted per map and the logic the testcase is employing for validation (verbose comments for the arithmetic is what I mean smile ).
            Hide
            Permalink
            Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12405486/5657-1.patch
            against trunk revision 765815.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            -1 contrib tests. The patch failed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/console

            This message is automatically generated.

            Show
            Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12405486/5657-1.patch against trunk revision 765815. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/205/console This message is automatically generated.
            Hide
            Permalink
            Chris Douglas added a comment -

            Adds some extra comments; no functional changes

            Show
            Chris Douglas added a comment - Adds some extra comments; no functional changes
            Hide
            Permalink
            Chris Douglas added a comment -

            I committed this.

            Show
            Chris Douglas added a comment - I committed this.
            Hide
            Permalink
            Hudson added a comment -

            Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/)
            . Validate data in TestReduceFetch to improve merge test coverage.

            Show
            Hudson added a comment - Integrated in Hadoop-trunk #821 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/821/ ) . Validate data in TestReduceFetch to improve merge test coverage.

              People

              • Assignee:
                Chris Douglas
                Reporter:
                Chris Douglas
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development