Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
    • Component/s: mrv2, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestMRApp is failing occasionally with this error:

      testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp): Expecting 2 more completion events for killed expected:<4> but was:<2>
      
      1. mapreduce4785.001.patch
        7 kB
        Haibo Chen
      2. mapreduce4785.prelim.patch
        7 kB
        Haibo Chen

        Activity

        Hide
        haibochen Haibo Chen added a comment -

        The cause of this problem is that the update of task/taskAttempt states is not in sync with that of TaskAttemptCompletionEvent of the job due to the async nature of the dispatcher dispatching the events. If the async dispatcher is delayed long enough, we will have cases where the task/taskAttempt states have changed, but the TaskAttemptCompletionEvent[] is still stale. We could wait for a maximum amount of time for the update of TaskAttemptCompletionEvents. Only in extreme cases when the update is delayed longer than our wait time, will this test fail and cause a false alarm. Please see my preliminary patch.

        Show
        haibochen Haibo Chen added a comment - The cause of this problem is that the update of task/taskAttempt states is not in sync with that of TaskAttemptCompletionEvent of the job due to the async nature of the dispatcher dispatching the events. If the async dispatcher is delayed long enough, we will have cases where the task/taskAttempt states have changed, but the TaskAttemptCompletionEvent[] is still stale. We could wait for a maximum amount of time for the update of TaskAttemptCompletionEvents. Only in extreme cases when the update is delayed longer than our wait time, will this test fail and cause a false alarm. Please see my preliminary patch.
        Hide
        haibochen Haibo Chen added a comment -

        Hi Jason Lowe, could you please see if this is an acceptable solution?

        Show
        haibochen Haibo Chen added a comment - Hi Jason Lowe , could you please see if this is an acceptable solution?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 36s trunk passed
        +1 compile 0m 21s trunk passed with JDK v1.8.0_72
        +1 compile 0m 24s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 47s trunk passed
        +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 20s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 19s the patch passed with JDK v1.8.0_72
        +1 javac 0m 19s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.7.0_95
        +1 javac 0m 22s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 27s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 58s the patch passed
        +1 javadoc 0m 14s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 17s the patch passed with JDK v1.7.0_95
        -1 unit 9m 59s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_72.
        +1 unit 10m 19s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 28s Patch does not generate ASF License warnings.
        37m 16s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789332/mapreduce4785.prelim.patch
        JIRA Issue MAPREDUCE-4785
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fb8158ea0923 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 480302b
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 36s trunk passed +1 compile 0m 21s trunk passed with JDK v1.8.0_72 +1 compile 0m 24s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 47s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 20s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 25s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_72 +1 javac 0m 19s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_95 +1 javac 0m 22s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 58s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 17s the patch passed with JDK v1.7.0_95 -1 unit 9m 59s hadoop-mapreduce-client-app in the patch failed with JDK v1.8.0_72. +1 unit 10m 19s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 28s Patch does not generate ASF License warnings. 37m 16s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789332/mapreduce4785.prelim.patch JIRA Issue MAPREDUCE-4785 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fb8158ea0923 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 480302b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6356/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        haibochen Haibo Chen added a comment -

        The failure is unrelated to this patch.

        Show
        haibochen Haibo Chen added a comment - The failure is unrelated to this patch.
        Hide
        rkanter Robert Kanter added a comment -

        The approach seems fine to me. A minor thing though:

        1. The latter Supplier}}s are still checking {{job1; they should be checking job2.
        Show
        rkanter Robert Kanter added a comment - The approach seems fine to me. A minor thing though: The latter Supplier}}s are still checking {{job1 ; they should be checking job2 .
        Hide
        haibochen Haibo Chen added a comment -

        Thanks for pointing that out. Have updated accordingly.

        Show
        haibochen Haibo Chen added a comment - Thanks for pointing that out. Have updated accordingly.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 43s trunk passed
        +1 compile 0m 19s trunk passed with JDK v1.8.0_72
        +1 compile 0m 22s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 43s trunk passed
        +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72
        +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 16s the patch passed with JDK v1.8.0_72
        +1 javac 0m 16s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.7.0_95
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 52s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95
        +1 unit 9m 11s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_72.
        +1 unit 9m 36s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 19s Patch does not generate ASF License warnings.
        32m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791077/mapreduce4785.001.patch
        JIRA Issue MAPREDUCE-4785
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d735c5f5523f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 67880cc
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6359/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6359/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 19s trunk passed with JDK v1.8.0_72 +1 compile 0m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 43s trunk passed +1 javadoc 0m 15s trunk passed with JDK v1.8.0_72 +1 javadoc 0m 17s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 23s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_72 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_95 +1 javac 0m 20s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 52s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_95 +1 unit 9m 11s hadoop-mapreduce-client-app in the patch passed with JDK v1.8.0_72. +1 unit 9m 36s hadoop-mapreduce-client-app in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 32m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791077/mapreduce4785.001.patch JIRA Issue MAPREDUCE-4785 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d735c5f5523f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67880cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6359/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6359/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        +1 LGTM.

        Will commit tomorrow if nobody has any other comments.

        Show
        rkanter Robert Kanter added a comment - +1 LGTM. Will commit tomorrow if nobody has any other comments.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        +1

        Show
        ajisakaa Akira Ajisaka added a comment - +1
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Haibo and others for reviews. Committed to trunk and branch-2!

        Show
        rkanter Robert Kanter added a comment - Thanks Haibo and others for reviews. Committed to trunk and branch-2!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9421 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9421/)
        MAPREDUCE-4785. TestMRApp occasionally fails (haibochen via rkanter) (rkanter: rev ff0ee84d77d9438f0954ae4e1497d63997bb7347)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9421 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9421/ ) MAPREDUCE-4785 . TestMRApp occasionally fails (haibochen via rkanter) (rkanter: rev ff0ee84d77d9438f0954ae4e1497d63997bb7347) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
        Hide
        leftnoteasy Wangda Tan added a comment -

        Backported this issue to branch-2.8 and branch-2.7, MAPREDUCE-6513 depends on it.

        Show
        leftnoteasy Wangda Tan added a comment - Backported this issue to branch-2.8 and branch-2.7, MAPREDUCE-6513 depends on it.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            haibochen Haibo Chen
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development