Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.3
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      TestJobImpl#testUnusableNodeTransition is flaky.

      2016-02-13 09:16:42 Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl
      2016-02-13 09:16:50 Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.324 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl
      2016-02-13 09:16:50 testUnusableNodeTransition(org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl) Time elapsed: 5.165 sec <<< FAILURE!
      2016-02-13 09:16:50 java.lang.AssertionError: expected:<SUCCEEDED> but was:<ERROR>
      2016-02-13 09:16:50 at org.junit.Assert.fail(Assert.java:88)
      2016-02-13 09:16:50 at org.junit.Assert.failNotEquals(Assert.java:743)
      2016-02-13 09:16:50 at org.junit.Assert.assertEquals(Assert.java:118)
      2016-02-13 09:16:50 at org.junit.Assert.assertEquals(Assert.java:144)
      2016-02-13 09:16:50 at org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.assertJobState(TestJobImpl.java:977)
      2016-02-13 09:16:50 at org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testUnusableNodeTransition(TestJobImpl.java:627)
      2016-02-13 09:16:50
      2016-02-13 09:16:50
      2016-02-13 09:16:50 Results :
      2016-02-13 09:16:50
      2016-02-13 09:16:50 Failed tests:
      2016-02-13 09:16:50 TestJobImpl.testUnusableNodeTransition:627->assertJobState:977 expected:<SUCCEEDED> but was:<ERROR>
      2016-02-13 09:16:50
      2016-02-13 09:16:50 Tests run: 17, Failures: 1, Errors: 0, Skipped: 0.

      Looking at the code, an JobUpdatedNodesEvent is handled by putting an TaskAttemptKill event on the async dispatcher queue and return immediately, but the event might not have been processed by the time all JobTaskEvents events are seen by the job (the jobTaskSucceeded events are handed to Job immediately without going through the dispatcher). Therefore, there is a slight chance that the job will see all three succeeded attempts and transition to Committing state before the taskAttemptKill event is handled by the dispatcher. Committing jobs will reject later JobTaskEvents received, transition to InternalError state and cause the test to fail.

        Activity

        Hide
        haibochen Haibo Chen added a comment -

        The change is limited to the test method

        Show
        haibochen Haibo Chen added a comment - The change is limited to the test method
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 7s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 37s trunk passed
        +1 compile 0m 18s trunk passed with JDK v1.8.0_77
        +1 compile 0m 23s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 15s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 44s trunk passed
        +1 javadoc 0m 14s trunk passed with JDK v1.8.0_77
        +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 15s the patch passed with JDK v1.8.0_77
        +1 javac 0m 15s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_95
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 24s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 53s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_77
        +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95
        +1 unit 9m 5s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77.
        +1 unit 9m 39s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        32m 36s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798978/mapreduce6675.001.patch
        JIRA Issue MAPREDUCE-6675
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 078ef8e71602 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6e6b6dd
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6435/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6435/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 7s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 0m 18s trunk passed with JDK v1.8.0_77 +1 compile 0m 23s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 44s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_77 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 15s the patch passed with JDK v1.8.0_77 +1 javac 0m 15s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 24s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 53s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_77 +1 javadoc 0m 15s the patch passed with JDK v1.7.0_95 +1 unit 9m 5s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_77. +1 unit 9m 39s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 32m 36s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798978/mapreduce6675.001.patch JIRA Issue MAPREDUCE-6675 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 078ef8e71602 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6e6b6dd Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6435/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6435/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        +1

        Show
        rkanter Robert Kanter added a comment - +1
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Haibo. Committed to trunk and branch-2!

        Show
        rkanter Robert Kanter added a comment - Thanks Haibo. Committed to trunk and branch-2!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9720 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9720/)
        MAPREDUCE-6675. TestJobImpl.testUnusableNode failed (haibochen via (rkanter: rev 9d3fcdfbb314c83ba6185e4ac8de649dad51a279)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9720 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9720/ ) MAPREDUCE-6675 . TestJobImpl.testUnusableNode failed (haibochen via (rkanter: rev 9d3fcdfbb314c83ba6185e4ac8de649dad51a279) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Hide
        eepayne Eric Payne added a comment -

        Thanks Haibo Chen for the fix. I have backported this to branch-2.8.

        Show
        eepayne Eric Payne added a comment - Thanks Haibo Chen for the fix. I have backported this to branch-2.8.

          People

          • Assignee:
            haibochen Haibo Chen
            Reporter:
            haibochen Haibo Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development