Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6831

Flaky test TestJobImpl.testKilledDuringKillAbort

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2, 2.8.2
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The test case TestJobImpl.testKilledDuringKillAbort() is flaky.

      Example of a failure:

      Error Message
      expected:<SETUP> but was:<RUNNING>
      
      Stack Trace
      java.lang.AssertionError: expected:<SETUP> but was:<RUNNING>
      	at org.junit.Assert.fail(Assert.java:88)
      	at org.junit.Assert.failNotEquals(Assert.java:743)
      	at org.junit.Assert.assertEquals(Assert.java:118)
      	at org.junit.Assert.assertEquals(Assert.java:144)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.assertJobState(TestJobImpl.java:978)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl.testKilledDuringKillAbort(TestJobImpl.java:516)
      
      Standard Output
      2016-12-12 00:26:29,724 INFO  [Thread-12] event.AsyncDispatcher (AsyncDispatcher.java:register(202)) - Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
      2016-12-12 00:26:29,729 INFO  [Thread-12] event.AsyncDispatcher (AsyncDispatcher.java:register(202)) - Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl$StubbedJob
      2016-12-12 00:26:29,729 INFO  [Thread-12] event.AsyncDispatcher (AsyncDispatcher.java:register(202)) - Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$2a4993a5
      2016-12-12 00:26:29,730 INFO  [Thread-12] event.AsyncDispatcher (AsyncDispatcher.java:register(202)) - Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$2a4993a5
      2016-12-12 00:26:29,730 INFO  [Thread-12] event.AsyncDispatcher (AsyncDispatcher.java:register(202)) - Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.yarn.event.EventHandler$$EnhancerByMockitoWithCGLIB$$2a4993a5
      2016-12-12 00:26:29,730 INFO  [Thread-12] impl.JobImpl (JobImpl.java:setup(1523)) - Adding job token for job_1234567890000_0001 to jobTokenSecretManager
      2016-12-12 00:26:29,731 WARN  [Thread-12] impl.JobImpl (JobImpl.java:setup(1529)) - Shuffle secret key missing from job credentials. Using job token secret as shuffle secret.
      2016-12-12 00:26:29,733 INFO  [Thread-12] impl.JobImpl (JobImpl.java:makeUberDecision(1294)) - Not uberizing job_1234567890000_0001 because: not enabled;
      2016-12-12 00:26:29,734 INFO  [Thread-12] impl.JobImpl (JobImpl.java:createMapTasks(1551)) - Input size for job job_1234567890000_0001 = 0. Number of splits = 2
      2016-12-12 00:26:29,734 INFO  [Thread-12] impl.JobImpl (JobImpl.java:createReduceTasks(1568)) - Number of reduces for job job_1234567890000_0001 = 1
      2016-12-12 00:26:29,734 INFO  [Thread-12] impl.JobImpl (JobImpl.java:handle(1006)) - job_1234567890000_0001Job Transitioned from NEW to INITED
      2016-12-12 00:26:29,736 INFO  [CommitterEvent Processor #0] commit.CommitterEventHandler (CommitterEventHandler.java:run(231)) - Processing the event EventType: JOB_SETUP
      2016-12-12 00:26:29,737 INFO  [Thread-12] impl.JobImpl (JobImpl.java:handle(1006)) - job_1234567890000_0001Job Transitioned from INITED to SETUP
      2016-12-12 00:26:29,738 INFO  [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1006)) - job_1234567890000_0001Job Transitioned from SETUP to RUNNING
      

      Reproduction: insert a Thread.sleep(50); after job.handle(new JobStartEvent(jobId));

      1. MAPREDUCE-6831-002.patch
        2 kB
        Peter Bacsko
      2. MAPREDUCE-6831-001.patch
        1 kB
        Peter Bacsko

        Activity

        Hide
        pbacsko Peter Bacsko added a comment -

        Simplest fix: not starting AsyncDispatcher does the job.

        Show
        pbacsko Peter Bacsko added a comment - Simplest fix: not starting AsyncDispatcher does the job.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 14m 8s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 27s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 37s trunk passed
        +1 javadoc 0m 16s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 20s the patch passed
        +1 javac 0m 20s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 22s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 46s the patch passed
        +1 javadoc 0m 13s the patch passed
        +1 unit 9m 3s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        29m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846336/MAPREDUCE-6831-001.patch
        JIRA Issue MAPREDUCE-6831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 502dea1da850 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / db490ec
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6863/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6863/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 8s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 37s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 20s the patch passed +1 javac 0m 20s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 46s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 9m 3s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 29m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846336/MAPREDUCE-6831-001.patch JIRA Issue MAPREDUCE-6831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 502dea1da850 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / db490ec Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6863/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6863/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the report and patch!

        It's a bit odd to create a dispatcher and then not initialize it. There needs to be a comment explaining why it's important to avoid the init/start, or someone later may think this was an accidental omission.

        Show
        jlowe Jason Lowe added a comment - Thanks for the report and patch! It's a bit odd to create a dispatcher and then not initialize it. There needs to be a comment explaining why it's important to avoid the init/start, or someone later may think this was an accidental omission.
        Hide
        pbacsko Peter Bacsko added a comment -

        OK, I added some comments.

        Show
        pbacsko Peter Bacsko added a comment - OK, I added some comments.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 12m 43s trunk passed
        +1 compile 0m 27s trunk passed
        +1 checkstyle 0m 17s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 0m 46s trunk passed
        +1 javadoc 0m 17s trunk passed
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21)
        +1 mvnsite 0m 28s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 54s the patch passed
        +1 javadoc 0m 15s the patch passed
        +1 unit 9m 28s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        28m 52s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847780/MAPREDUCE-6831-002.patch
        JIRA Issue MAPREDUCE-6831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0a877f3e2692 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7ee8be1
        Default Java 1.8.0_111
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6868/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6868/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 43s trunk passed +1 compile 0m 27s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 46s trunk passed +1 javadoc 0m 17s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed +1 checkstyle 0m 15s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: The patch generated 0 new + 20 unchanged - 1 fixed = 20 total (was 21) +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 54s the patch passed +1 javadoc 0m 15s the patch passed +1 unit 9m 28s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 28m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847780/MAPREDUCE-6831-002.patch JIRA Issue MAPREDUCE-6831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0a877f3e2692 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7ee8be1 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6868/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6868/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        +1 lgtm. Committing this.

        Show
        jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Peter! I committed this to trunk, branch-2, and branch-2.8.

        Show
        jlowe Jason Lowe added a comment - Thanks, Peter! I committed this to trunk, branch-2, and branch-2.8.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11128/)
        MAPREDUCE-6831. Flaky test TestJobImpl.testKilledDuringKillAbort. (jlowe: rev 78b487bde175544ebe40e4dafab35569baa1d79e)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11128/ ) MAPREDUCE-6831 . Flaky test TestJobImpl.testKilledDuringKillAbort. (jlowe: rev 78b487bde175544ebe40e4dafab35569baa1d79e) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            pbacsko Peter Bacsko
            Reporter:
            pbacsko Peter Bacsko
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development