Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5422 [Umbrella] Fix invalid state transitions in MRAppMaster
  3. MAPREDUCE-5400

MRAppMaster throws InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED for JobImpl

    Details

    • Type: Sub-task Sub-task
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0.5-alpha
    • Fix Version/s: None
    • Component/s: applicationmaster
    • Labels:
      None
    • Target Version/s:

      Description

      Step 1: Install cluster with HDFS , MR
      Step 2: Execute a job
      Step 3: Issue a kill task attempt for which the task has got completed.

      Rex@HOST-10-18-91-55:~/NodeAgentTmpDir/installations/hadoop-2.0.5.tar/hadoop-2.0.5/bin> ./mapred job -kill-task attempt_1373875322959_0032_m_000000_0
      No GC_PROFILE is given. Defaults to medium.
      13/07/15 14:46:32 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
      13/07/15 14:46:32 INFO proxy.ResourceManagerProxies: HA Proxy Creation with xface : interface org.apache.hadoop.yarn.api.ClientRMProtocol
      13/07/15 14:46:33 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
      Killed task attempt_1373875322959_0032_m_000000_0

      Observation:
      ===========
      1. task state has been transitioned from SUCCEEDED to SCHEDULED
      2. For a Succeeded attempt , when client issues Kill , then the client is notified as killed for a succeeded attempt.
      3. Launched second task_attempt which is succeeded and then killed later on client request.
      4. Even after the job state transitioned from SUCCEEDED to ERROR , on UI the state is succeeded

      Issue :
      =====
      1. Client has been notified that the atttempt is killed , but acutually the attempt is succeeded and the same is displayed in JHS UI.
      2. At App master InvalidStateTransitonException is thrown .
      3. At client side and JHS job has exited with state Finished/succeeded ,At RM side the state is Finished/Failed.

      AM Logs:
      ========
      2013-07-15 14:46:25,461 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1373875322959_0032_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCEEDED
      2013-07-15 14:46:25,468 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1373875322959_0032_m_000000_0
      2013-07-15 14:46:25,470 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1373875322959_0032_m_000000 Task Transitioned from RUNNING to SUCCEEDED
      2013-07-15 14:46:33,810 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1373875322959_0032_m_000000 Task Transitioned from SUCCEEDED to SCHEDULED
      2013-07-15 14:46:37,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1373875322959_0032_m_000000_1
      2013-07-15 14:46:37,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1373875322959_0032_m_000000 Task Transitioned from RUNNING to SUCCEEDED
      2013-07-15 14:46:37,345 ERROR [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event at current state
      org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED
      at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
      at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
      at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:866)
      at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:128)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1095)
      at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1091)
      at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
      at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
      at java.lang.Thread.run(Thread.java:662)

      1. MAPREDUCE-5400-1.patch
        2 kB
        Devaraj K
      2. MAPREDUCE-5400.patch
        2 kB
        Devaraj K

        Activity

        Hide
        Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 37s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 javadoc 9m 38s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 35s The applied patch generated 1 new checkstyle issues (total was 107, now 107).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 0m 57s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 9m 56s Tests passed in hadoop-mapreduce-client-app.
            45m 49s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732822/MAPREDUCE-5400-1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / ffbb574
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
        hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/console

        This message was automatically generated.

        Show
        Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 35s The applied patch generated 1 new checkstyle issues (total was 107, now 107). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 57s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 9m 56s Tests passed in hadoop-mapreduce-client-app.     45m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732822/MAPREDUCE-5400-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ffbb574 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5734/console This message was automatically generated.
        Hide
        Devaraj K added a comment -

        Updated patch for this issue.

        Show
        Devaraj K added a comment - Updated patch for this issue.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12593329/MAPREDUCE-5400.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3875//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3875//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12593329/MAPREDUCE-5400.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3875//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3875//console This message is automatically generated.
        Hide
        Devaraj K added a comment -

        Attached the patch to fix this issue. Please review this.

        Show
        Devaraj K added a comment - Attached the patch to fix this issue. Please review this.
        Hide
        Devaraj K added a comment -

        It is an issue from M/R App Master, moving it to mapreduce project.

        Show
        Devaraj K added a comment - It is an issue from M/R App Master, moving it to mapreduce project.

          People

          • Assignee:
            Devaraj K
            Reporter:
            J.Andreina
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:

              Development