Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5155

Race condition in test case TestFetchFailure cause it to fail

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: test
    • Labels:
      None
    • Environment:

      Suse x86_64 GNU/Linux
      Java(TM) SE Runtime Environment (build 1.6.0_32-b05

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I run into this once: testFetchFailureWithRecovery(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure): Num completion events not correct expected:<1> but was:<0>

      There is a race condition between job.getTaskAttemptCompletionEvents and dealing with JOB_TASK_ATTEMPT_COMPLETED event.
      If job.getTaskAttemptCompletionEvents invoked because of task in SUCCEEDED state ,but before JOB_TASK_ATTEMPT_COMPLETED event scheduled,the test case will fail.

        Activity

        Hide
        nemon Nemon Lou added a comment -

        Logs are uploaded

        Show
        nemon Nemon Lou added a comment - Logs are uploaded
        Hide
        nemon Nemon Lou added a comment -

        Another test case failed with the same reason:
        Failed tests: testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp): Expecting 2 more completion events for killed expected:<4> but was:<2>

        These lines of code has race condition issue:

        app.waitForState(task1Attempt, TaskAttemptState.KILLED);
        app.waitForState(task2Attempt, TaskAttemptState.KILLED);

        events = job.getTaskAttemptCompletionEvents(0, 100);
        Assert.assertEquals("Expecting 2 more completion events for killed", 4,
        events.length);

        Show
        nemon Nemon Lou added a comment - Another test case failed with the same reason: Failed tests: testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp): Expecting 2 more completion events for killed expected:<4> but was:<2> These lines of code has race condition issue: app.waitForState(task1Attempt, TaskAttemptState.KILLED); app.waitForState(task2Attempt, TaskAttemptState.KILLED); events = job.getTaskAttemptCompletionEvents(0, 100); Assert.assertEquals("Expecting 2 more completion events for killed", 4, events.length);
        Hide
        haibochen Haibo Chen added a comment -

        TestMRApp.testUpdatedNodes() has been fixed in MAPREDUCE-4785

        Show
        haibochen Haibo Chen added a comment - TestMRApp.testUpdatedNodes() has been fixed in MAPREDUCE-4785
        Hide
        haibochen Haibo Chen added a comment -

        Upload a patch that contains the same fix in in MAPREDUCE-4785

        Show
        haibochen Haibo Chen added a comment - Upload a patch that contains the same fix in in MAPREDUCE-4785
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 52s trunk passed
        +1 compile 0m 22s trunk passed
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 30s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 37s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 42s the patch passed
        +1 javadoc 0m 13s the patch passed
        +1 unit 8m 55s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        21m 44s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842209/MAPREDUCE-5155.01.patch
        JIRA Issue MAPREDUCE-5155
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2b499480ce91 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9f5d2c4
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6835/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6835/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 52s trunk passed +1 compile 0m 22s trunk passed +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 30s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 37s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 42s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 8m 55s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 21m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842209/MAPREDUCE-5155.01.patch JIRA Issue MAPREDUCE-5155 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2b499480ce91 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f5d2c4 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6835/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6835/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Looks good to me. +1. I'll commit tomorrow unless there are other comments.

        Show
        templedf Daniel Templeton added a comment - Looks good to me. +1. I'll commit tomorrow unless there are other comments.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks for the patch, Haibo Chen. Committed to trunk. If you want it to go into branch-2, please provide a branch-2 patch. (Java 7 wants the var used in the anon inner class to be final.)

        Show
        templedf Daniel Templeton added a comment - Thanks for the patch, Haibo Chen . Committed to trunk. If you want it to go into branch-2, please provide a branch-2 patch. (Java 7 wants the var used in the anon inner class to be final.)
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11057 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11057/)
        MAPREDUCE-5155. Race condition in test case TestFetchFailure cause it to (templedf: rev 95c2c24cd0007ece9d7cd0fbe4c5932828e639d4)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11057 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11057/ ) MAPREDUCE-5155 . Race condition in test case TestFetchFailure cause it to (templedf: rev 95c2c24cd0007ece9d7cd0fbe4c5932828e639d4) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java

          People

          • Assignee:
            haibochen Haibo Chen
            Reporter:
            nemon Nemon Lou
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development