Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: contrib/gridmix
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed errors that caused occasional failure of TestGridmixSubmission, and additionally refactored some Gridmix code.

      Description

      TestGridmixSubmission fails sometimes with following error :
      Mismatched output bytes 4547848/4561267
      junit.framework.AssertionFailedError: Mismatched output bytes 4547848/4561267
      at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.check(TestGridmixSubmission.java:231)
      at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$TestMonitor.verify(TestGridmixSubmission.java:140)
      at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission$DebugGridmix.checkMonitor(TestGridmixSubmission.java:263)
      at org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.testSubmit(TestGridmixSubmission.java:297)

      1. M1124-0.patch
        5 kB
        Chris Douglas
      2. M1124-1.patch
        5 kB
        Chris Douglas
      3. M1124-2.patch
        6 kB
        Chris Douglas
      4. M1124-3.patch
        136 kB
        Chris Douglas
      5. M1124-4.patch
        142 kB
        Chris Douglas
      6. M1124-5.patch
        142 kB
        Chris Douglas
      7. M1124-y20.patch
        1.04 MB
        Chris Douglas
      8. M1124-y20-1.patch
        1.04 MB
        Chris Douglas
      9. TEST-org.apache.hadoop.mapred.gridmix.TestGridmixSubmission.txt
        187 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Amareshwari Sriramadasu added a comment -

        Test failure log attached

        Show
        Amareshwari Sriramadasu added a comment - Test failure log attached
        Hide
        Chris Douglas added a comment -

        Fixed two bugs:

        • The adjustment for length encoding out of the map was in the wrong place
        • The reduce input bytes are taken from the shuffle byte counter, which adds at least 2 bytes for encoding record lengths
        Show
        Chris Douglas added a comment - Fixed two bugs: The adjustment for length encoding out of the map was in the wrong place The reduce input bytes are taken from the shuffle byte counter, which adds at least 2 bytes for encoding record lengths
        Hide
        Hong Tang added a comment -

        patch looks good. Very minor nit, the following statements seems to be useless now.

        +      valsize = Math.max(0, valsize);
        
        Show
        Hong Tang added a comment - patch looks good. Very minor nit, the following statements seems to be useless now. + valsize = Math.max(0, valsize);
        Hide
        Chris Douglas added a comment -

        Good point. Removed.

        Show
        Chris Douglas added a comment - Good point. Removed.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423088/M1124-0.patch
        against trunk revision 829281.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423088/M1124-0.patch against trunk revision 829281. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/91/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        More adjustments for shuffle bytes (instead of assuming 2 bytes/record, approximate w/ average record size and key length)

        Show
        Chris Douglas added a comment - More adjustments for shuffle bytes (instead of assuming 2 bytes/record, approximate w/ average record size and key length)
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423106/M1124-2.patch
        against trunk revision 829312.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423106/M1124-2.patch against trunk revision 829312. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/92/console This message is automatically generated.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Is this good for commit? If so, can someone please commit this? It's failing a lot of builds. Thanks!

        Show
        Vinod Kumar Vavilapalli added a comment - Is this good for commit? If so, can someone please commit this? It's failing a lot of builds. Thanks!
        Hide
        Chris Douglas added a comment -
        • Refactored GridmixJob, et al.
        • TestGridmixSubmission passes reliably, as do new component tests.
        Show
        Chris Douglas added a comment - Refactored GridmixJob, et al. TestGridmixSubmission passes reliably, as do new component tests.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12426629/M1124-3.patch
        against trunk revision 886042.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 14 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426629/M1124-3.patch against trunk revision 886042. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/159/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        Updated patch. Thanks to Hong and Dick for their review comments.

        Show
        Chris Douglas added a comment - Updated patch. Thanks to Hong and Dick for their review comments.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427400/M1124-4.patch
        against trunk revision 888431.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 14 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427400/M1124-4.patch against trunk revision 888431. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 14 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/305/console This message is automatically generated.
        Hide
        Hong Tang added a comment -

        I like the latest patch. +1. A few very minor nits:

        • GridmixJob.java : unused imports: java.util.Comparator; org.apache.hadoop.fs.FSDataInputStream; org.apache.hadoop.io.IOUtils; org.apache.hadoop.io.IntWritable; org.apache.hadoop.io.Writable;
        • JobMonitor.java : javadoc for "public JobMonitor(int pollDelay, TimeUnit unit)" has an extra @param element.
        • ReadRecordFactory.java : javadoc for "public ReadRecordFactory(long targetBytes, long targetRecords, InputStream src, Configuration conf)" has an extra @param element.
        Show
        Hong Tang added a comment - I like the latest patch. +1. A few very minor nits: GridmixJob.java : unused imports: java.util.Comparator; org.apache.hadoop.fs.FSDataInputStream; org.apache.hadoop.io.IOUtils; org.apache.hadoop.io.IntWritable; org.apache.hadoop.io.Writable; JobMonitor.java : javadoc for "public JobMonitor(int pollDelay, TimeUnit unit)" has an extra @param element. ReadRecordFactory.java : javadoc for "public ReadRecordFactory(long targetBytes, long targetRecords, InputStream src, Configuration conf)" has an extra @param element.
        Hide
        Chris Douglas added a comment -

        Fixed javadoc (odd that it doesn't warn) and removed unused imports.

        Show
        Chris Douglas added a comment - Fixed javadoc (odd that it doesn't warn) and removed unused imports.
        Hide
        Chris Douglas added a comment -

        I committed this.

        Show
        Chris Douglas added a comment - I committed this.
        Hide
        Chris Douglas added a comment -

        Patch for y20.

        Show
        Chris Douglas added a comment - Patch for y20.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )

          People

          • Assignee:
            Chris Douglas
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development