Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-7240

Exception ' Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER' cause job error

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      log in appmaster

      2019-09-03 17:18:43,090 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures for output of task attempt: attempt_1566552310686_260041_m_000052_0 ... raising fetch failure to map
      2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures for output of task attempt: attempt_1566552310686_260041_m_000049_0 ... raising fetch failure to map
      2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures for output of task attempt: attempt_1566552310686_260041_m_000051_0 ... raising fetch failure to map
      2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures for output of task attempt: attempt_1566552310686_260041_m_000050_0 ... raising fetch failure to map
      2019-09-03 17:18:43,091 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures for output of task attempt: attempt_1566552310686_260041_m_000053_0 ... raising fetch failure to map
      2019-09-03 17:18:43,092 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1566552310686_260041_m_000052_0 transitioned from state SUCCEEDED to FAILED, event type is TA_TOO_MANY_FETCH_FAILURE and nodeId=yarn095:45454
      2019-09-03 17:18:43,092 ERROR [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this event at current state for attempt_1566552310686_260041_m_000049_0
      org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER
      	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
      	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1206)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:146)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1458)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1450)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
      	at java.lang.Thread.run(Thread.java:745)
      2019-09-03 17:18:43,093 ERROR [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this event at current state for attempt_1566552310686_260041_m_000051_0
      org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER
      	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
      	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1206)
      	at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:146)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1458)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1450)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
      	at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
      	at java.lang.Thread.run(Thread.java:745)
      2019-09-03 17:18:43,093 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1566552310686_260041_m_000050_0 transitioned from state SUCCEEDED to FAILED, event type is TA_TOO_MANY_FETCH_FAILURE and nodeId=yarn095:45454
      2019-09-03 17:18:43,093 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1566552310686_260041_m_000053_0 transitioned from state SUCCEEDED to FAILED, event type is TA_TOO_MANY_FETCH_FAILURE and nodeId=yarn095:45454
      2019-09-03 17:18:43,094 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1566552310686_260041_m_000052 Task Transitioned from SUCCEEDED to SCHEDULED
      2019-09-03 17:18:43,096 FATAL [IPC Server handler 27 on 35972] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1566552310686_260041_r_000005_0 - exited : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#22
      	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
      	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1961)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
      Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
      	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:367)
      	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:289)
      	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:355)
      	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
      
      2019-09-03 17:18:43,096 INFO [IPC Server handler 27 on 35972] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1566552310686_260041_r_000005_0: Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#22
      	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
      	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
      	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1961)
      	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
      Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
      	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:367)
      	at org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:289)
      	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:355)
      	at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
      
      2019-09-03 17:18:43,097 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1566552310686_260041Job Transitioned from RUNNING to ERROR
      2019-09-03 17:18:43,099 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job
      
      

        

      nodemanager's  log is like same with log in  MAPREDUCE-6869.

      the code in TaskAttemptImpl indicate the Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER cause the job state turn into error; what i confused is 

      1. what cause the appmater  handle the TA_TOO_MANY_FETCH_FAILURE  event on SUCCESS_FINISHING_CONTAINER,illegal event on this state.  but some other can successfully transitioned from state SUCCEEDED to FAILED on TA_TOO_MANY_FETCH_FAILURE  event.
      2. restart the nodemanager would solve the error in nm; the shuffle error would fix too. what cause this phenomenon.

      Correct me if I am wrong.

        Attachments

        1. application_1566552310686_260041.log
          20.76 MB
          luhuachao
        2. MAPREDUCE-7240-001.patch
          4 kB
          Peter Bacsko
        3. MAPREDUCE-7240-002.patch
          4 kB
          Prabhu Joseph
        4. MAPREDUCE-7240-branch-3.1.001.patch
          4 kB
          Peter Bacsko
        5. MAPREDUCE-7240-branch-3.2.001.patch
          4 kB
          Peter Bacsko
        6. MAPREDUCE-7240-branch-3.2.001.patch
          4 kB
          Peter Bacsko

          Issue Links

            Activity

              People

              • Assignee:
                Huachao luhuachao
                Reporter:
                Huachao luhuachao
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: