Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This test has timed out in a number of Hudson builds.

      1. mapreduce-2284.txt
        3 kB
        Todd Lipcon
      2. mapreduce-2284-help-diagnose.txt
        2 kB
        Todd Lipcon

        Activity

        Hide
        tlipcon Todd Lipcon added a comment -

        I'd like to commit this patch to help us track down this failure (I can't reproduce it locally even looping the test overnight)

        Show
        tlipcon Todd Lipcon added a comment - I'd like to commit this patch to help us track down this failure (I can't reproduce it locally even looping the test overnight)
        Hide
        cos Konstantin Boudnik added a comment -

        +1 - totally makes sense! Let's commit it.

        Show
        cos Konstantin Boudnik added a comment - +1 - totally makes sense! Let's commit it.
        Hide
        tomwhite Tom White added a comment -

        I couldn't reproduce this locally either. I tried the diagnostic patch and it still allows the test to run for me locally, so I've committed this to see what Hudson makes of it.

        Show
        tomwhite Tom White added a comment - I couldn't reproduce this locally either. I tried the diagnostic patch and it still allows the test to run for me locally, so I've committed this to see what Hudson makes of it.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #614 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/614/)
        Diagnostic change to TestLocalRunner for MAPREDUCE-2284.

        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #614 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/614/ ) Diagnostic change to TestLocalRunner for MAPREDUCE-2284 .
        Hide
        tlipcon Todd Lipcon added a comment -

        Looking at the consoleText of one of the timeouts, I see that the test makes progress, just really slowly. I wonder if this is due to clover? The test does a lot of busy loops, maybe when clover instrumentation is on, it just runs too slowly? I'll give a try later tonight.

        Show
        tlipcon Todd Lipcon added a comment - Looking at the consoleText of one of the timeouts, I see that the test makes progress, just really slowly. I wonder if this is due to clover? The test does a lot of busy loops, maybe when clover instrumentation is on, it just runs too slowly? I'll give a try later tonight.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12470801/mapreduce-2284-help-diagnose.txt
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/36//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470801/mapreduce-2284-help-diagnose.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/36//console This message is automatically generated.
        Hide
        tomwhite Tom White added a comment -

        > I wonder if this is due to clover? The test does a lot of busy loops, maybe when clover instrumentation is on, it just runs too slowly?

        Looks like this is exactly what is happening: https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/606/testReport/org.apache.hadoop.mapreduce/TestLocalRunner/testMultiMaps/

        We could fix this by removing the busy loops, replacing them with loops with different sleep times.

        Show
        tomwhite Tom White added a comment - > I wonder if this is due to clover? The test does a lot of busy loops, maybe when clover instrumentation is on, it just runs too slowly? Looks like this is exactly what is happening: https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/606/testReport/org.apache.hadoop.mapreduce/TestLocalRunner/testMultiMaps/ We could fix this by removing the busy loops, replacing them with loops with different sleep times.
        Hide
        tlipcon Todd Lipcon added a comment -

        Reworked the test to just sleep different amounts in the different mappers. Runs in 26 seconds now even with clover enabled on my box.

        Show
        tlipcon Todd Lipcon added a comment - Reworked the test to just sleep different amounts in the different mappers. Runs in 26 seconds now even with clover enabled on my box.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472796/mapreduce-2284.txt
        against trunk revision 1078656.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/122//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/122//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/122//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472796/mapreduce-2284.txt against trunk revision 1078656. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/122//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/122//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/122//console This message is automatically generated.
        Hide
        tomwhite Tom White added a comment -

        +1 Do you want to remove the diagnostic code too?

        Show
        tomwhite Tom White added a comment - +1 Do you want to remove the diagnostic code too?
        Hide
        tlipcon Todd Lipcon added a comment -

        nah, I don't see any harm in leaving it in, do you? Should this ever start failing again in the future it's a useful thing to have.

        Show
        tlipcon Todd Lipcon added a comment - nah, I don't see any harm in leaving it in, do you? Should this ever start failing again in the future it's a useful thing to have.
        Hide
        tomwhite Tom White added a comment -

        OK, but it would be good to remove once we're sure that it's no longer a problem.

        Show
        tomwhite Tom White added a comment - OK, but it would be good to remove once we're sure that it's no longer a problem.
        Hide
        tlipcon Todd Lipcon added a comment -

        Committed to trunk and branch. Fingers crossed for a green build Thanks Tom

        Show
        tlipcon Todd Lipcon added a comment - Committed to trunk and branch. Fingers crossed for a green build Thanks Tom
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/)

        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/ )
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #38 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/38/)

        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #38 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/38/ )
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            tlipcon Todd Lipcon
            Reporter:
            tlipcon Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development