Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5631

TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 0.23.10, 2.3.0
    • Fix Version/s: 3.0.0, 2.3.0
    • Component/s: None
    • Labels:

      Description

      Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order.

      1. MAPREDUCE-5631.patch
        1 kB
        Jonathan Eagles
      2. MAPREDUCE-5631.patch
        6 kB
        Jonathan Eagles

        Activity

        Jonathan Eagles created issue -
        Jonathan Eagles made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-5631.patch [ 12614447 ]
        Jonathan Eagles made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614447/MAPREDUCE-5631.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

        org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4214//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4214//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614447/MAPREDUCE-5631.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4214//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4214//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        TestRMContainerAllocator fails due to MAPREDUCE-5427

        Show
        Jonathan Eagles added a comment - TestRMContainerAllocator fails due to MAPREDUCE-5427
        Hide
        Mit Desai added a comment -

        lgtm. +1 (non binding)

        Show
        Mit Desai added a comment - lgtm. +1 (non binding)
        Hide
        Jason Lowe added a comment -

        Could you elaborate on how the configuration is leaking between tests? The test cases all construct a new Configuration object and call setConf with it, so I'm not seeing how the configuration setting from one test is leaking into another. Seems more likely the properties being explicitly set by the patch were being accidentally picked up from a mapred-site.xml file in some environments which would break the test. Just want to make sure I understand the nature of the fix.

        Minor nit in the patch: the order of the last two arguments in the following code should be flipped since the Assert.assertEquals calling convention is expected, actual instead of actual, expected:

            Assert.assertEquals("Only 1 try was expected but was : "
              + this.notificationCount, this.notificationCount, 1);
        
        Show
        Jason Lowe added a comment - Could you elaborate on how the configuration is leaking between tests? The test cases all construct a new Configuration object and call setConf with it, so I'm not seeing how the configuration setting from one test is leaking into another. Seems more likely the properties being explicitly set by the patch were being accidentally picked up from a mapred-site.xml file in some environments which would break the test. Just want to make sure I understand the nature of the fix. Minor nit in the patch: the order of the last two arguments in the following code should be flipped since the Assert.assertEquals calling convention is expected, actual instead of actual, expected : Assert.assertEquals( "Only 1 try was expected but was : " + this .notificationCount, this .notificationCount, 1);
        Hide
        Jonathan Eagles added a comment -

        Good thought, Jason. After further investigation, the failing test is using a Configuration object. If it runs first, it passes since none of the

        {mapred,yarn}

        -

        {default,site}

        .xml and assumes the source code default retry interval of 5 seconds. If it runs after any of the test that run a job, it fails since the first use of JobConf will add the default and site files to the default resources of the configuration class. The assumption of falling back to the source code default of 5 seconds is no longer valid, since it is picking of the mapred-default.xml value of 1 second.

        I will change this test class to use the jobconf object to avoid any test order dependencies and add the config settings explicit to avoid changes to the default.xml causing the test to fail again.

        Show
        Jonathan Eagles added a comment - Good thought, Jason. After further investigation, the failing test is using a Configuration object. If it runs first, it passes since none of the {mapred,yarn} - {default,site} .xml and assumes the source code default retry interval of 5 seconds. If it runs after any of the test that run a job, it fails since the first use of JobConf will add the default and site files to the default resources of the configuration class. The assumption of falling back to the source code default of 5 seconds is no longer valid, since it is picking of the mapred-default.xml value of 1 second. I will change this test class to use the jobconf object to avoid any test order dependencies and add the config settings explicit to avoid changes to the default.xml causing the test to fail again.
        Jonathan Eagles made changes -
        Attachment MAPREDUCE-5631.patch [ 12614696 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12614696/MAPREDUCE-5631.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

        org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4217//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4217//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12614696/MAPREDUCE-5631.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4217//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4217//console This message is automatically generated.
        Hide
        Jonathan Eagles added a comment -

        Test failures not related to this patch
        org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator is handled by MAPREDUCE-5632

        Show
        Jonathan Eagles added a comment - Test failures not related to this patch org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator is handled by MAPREDUCE-5632
        Hide
        Chen He added a comment -

        +1, non binding

        Show
        Chen He added a comment - +1, non binding
        Hide
        Jason Lowe added a comment -

        +1, latest patch looks good. Committing this.

        Show
        Jason Lowe added a comment - +1, latest patch looks good. Committing this.
        Hide
        Jason Lowe added a comment -

        Thanks, Jon! I committed this to trunk and branch-2.

        Patch did not apply to branch-0.23, so if you are interested in targeting that release please post an updated patch.

        Show
        Jason Lowe added a comment - Thanks, Jon! I committed this to trunk and branch-2. Patch did not apply to branch-0.23, so if you are interested in targeting that release please post an updated patch.
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 3.0.0 [ 12320355 ]
        Fix Version/s 2.3.0 [ 12324590 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4780 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4780/)
        MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4780 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4780/ ) MAPREDUCE-5631 . TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #399 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/399/)
        MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #399 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/399/ ) MAPREDUCE-5631 . TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1590 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1590/)
        MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1590 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1590/ ) MAPREDUCE-5631 . TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1616 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1616/)
        MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266)

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1616 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1616/ ) MAPREDUCE-5631 . TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1544266 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
        Arun C Murthy made changes -
        Affects Version/s 2.3.0 [ 12325257 ]
        Affects Version/s 2.4.0 [ 12324590 ]
        Fix Version/s 2.3.0 [ 12325257 ]
        Fix Version/s 2.4.0 [ 12324590 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jonathan Eagles
            Reporter:
            Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development