Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4924

flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 1.2.0, 2.0.3-alpha
    • Component/s: mrv1
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I occasionally get a failure like this on org.apache.hadoop.mapred.TestClusterMRNotification.testMR

      junit.framework.AssertionFailedError: expected:<6> but was:<4>
      	at junit.framework.Assert.fail(Assert.java:47)
      	at junit.framework.Assert.failNotEquals(Assert.java:283)
      	at junit.framework.Assert.assertEquals(Assert.java:64)
      	at junit.framework.Assert.assertEquals(Assert.java:195)
      	at junit.framework.Assert.assertEquals(Assert.java:201)
      	at org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:184)
      	...
      

      It looks like a race condition:

          // run a job with FAILED status
          System.out.println(UtilsForTests.runJobFail(this.createJobConf(), inDir,
                                                      outDir).getID());
          Thread.sleep(2000);
          assertEquals(6, NotificationServlet.counter);
          assertEquals(0, NotificationServlet.failureCounter);
      

      Instead of sleeping for 2 seconds, we should keep checking the counter and fail after a timeout. There's a couple of similar places in the test that should be fixed too.

      1. MAPREDUCE-4924_branch-1.patch
        2 kB
        Robert Kanter
      2. MAPREDUCE-4924_branch-1.patch
        2 kB
        Robert Kanter
      3. MAPREDUCE-4924.patch
        3 kB
        Robert Kanter
      4. MAPREDUCE-4924.patch
        2 kB
        Robert Kanter

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1315/)
        MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1315/ ) MAPREDUCE-4924 . flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1287 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1287/)
        MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1287 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1287/ ) MAPREDUCE-4924 . flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/98/)
        MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/98/ ) MAPREDUCE-4924 . flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Hide
        Alejandro Abdelnur added a comment -

        Dah!!. Thanks Robert. Committed to trunk, branch-1 & branch-2.

        Show
        Alejandro Abdelnur added a comment - Dah!!. Thanks Robert. Committed to trunk, branch-1 & branch-2.
        Hide
        Robert Kanter added a comment -

        NotificationTestCase is abstract; its the parent class for the TestClusterMRNotification and TestLocalMRNotification tests. If you run those two test classes, it should work.

        Show
        Robert Kanter added a comment - NotificationTestCase is abstract; its the parent class for the TestClusterMRNotification and TestLocalMRNotification tests. If you run those two test classes, it should work.
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Robert. I've just committed this to trunk and branch-2. The patch for branch-1 seems to have an issue and fails with:

        Testsuite: org.apache.hadoop.mapred.NotificationTestCase
        Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.005 sec
        
        Testcase: warning took 0.002 sec
        	FAILED
        Class org.apache.hadoop.mapred.NotificationTestCase has no public constructor TestCase(String name) or TestCase()
        junit.framework.AssertionFailedError: Class org.apache.hadoop.mapred.NotificationTestCase has no public constructor TestCase(String name) or TestCase()
        

        Would you please check this?

        Show
        Alejandro Abdelnur added a comment - Thanks Robert. I've just committed this to trunk and branch-2. The patch for branch-1 seems to have an issue and fails with: Testsuite: org.apache.hadoop.mapred.NotificationTestCase Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 0.005 sec Testcase: warning took 0.002 sec FAILED Class org.apache.hadoop.mapred.NotificationTestCase has no public constructor TestCase( String name) or TestCase() junit.framework.AssertionFailedError: Class org.apache.hadoop.mapred.NotificationTestCase has no public constructor TestCase( String name) or TestCase() Would you please check this?
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3247 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3247/)
        MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3247 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3247/ ) MAPREDUCE-4924 . flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. (rkanter via tucu) (Revision 1433753) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433753 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
        Hide
        Robert Kanter added a comment -

        The test failure is unrelated. It fails without the patch too.

        Show
        Robert Kanter added a comment - The test failure is unrelated. It fails without the patch too.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12565019/MAPREDUCE-4924.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

        org.apache.hadoop.mapred.TestYARNRunner

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3245//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3245//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565019/MAPREDUCE-4924.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.TestYARNRunner +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3245//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3245//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        +1 pending jenkins

        Show
        Alejandro Abdelnur added a comment - +1 pending jenkins
        Hide
        Robert Kanter added a comment -

        Uploaded new version of the patches that use volatile and 50ms

        Show
        Robert Kanter added a comment - Uploaded new version of the patches that use volatile and 50ms
        Hide
        Alejandro Abdelnur added a comment -

        The the following to variables should be 'volatile':

        public static int counter = 0;
        public static int failureCounter = 0;

        Instead looping/sleeping 1000ms I'do it for 50ms, as this is done 4 times we are cutting an avg of 2 secs (max 4secs) the test.

        Show
        Alejandro Abdelnur added a comment - The the following to variables should be 'volatile': public static int counter = 0; public static int failureCounter = 0; Instead looping/sleeping 1000ms I'do it for 50ms, as this is done 4 times we are cutting an avg of 2 secs (max 4secs) the test.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12563845/MAPREDUCE-4924.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3213//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3213//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12563845/MAPREDUCE-4924.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3213//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3213//console This message is automatically generated.
        Hide
        Robert Kanter added a comment -

        It will now keep checking every second for up to 30 seconds

        Show
        Robert Kanter added a comment - It will now keep checking every second for up to 30 seconds
        Hide
        Robert Kanter added a comment -

        I probably won't get to this for a while, so if somebody else wants to do it feel free to reassign it to yourself.

        Show
        Robert Kanter added a comment - I probably won't get to this for a while, so if somebody else wants to do it feel free to reassign it to yourself.

          People

          • Assignee:
            Robert Kanter
            Reporter:
            Robert Kanter
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development