Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2995

MR AM crashes when a container-launch hangs on a faulty NM

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      AM tries to launch containers on a faulty node which blocks several/all of the StartContainer requests. Eventually, RM expires the container-allocations, informs the AM about container-expiry. But AM crashes with an INTERNAL_ERROR as the event is unexpected.

      11/09/12 14:11:38 ERROR impl.TaskAttemptImpl: Can't handle this event at current state
      org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_CONTAINER_COMPLETED at ASSIGNED
              at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:297)
              at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:39)
              at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:439)
              at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:903)
              at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:127)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:543)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:536)
              at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:113)
              at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
              at java.lang.Thread.run(Thread.java:619)
      

      Found this on a big cluster where Karam Singh was trying to get sort running.

        Attachments

        1. MAPREDUCE-2995-20110913.txt
          8 kB
          Vinod Kumar Vavilapalli

          Issue Links

            Activity

              People

              • Assignee:
                vinodkv Vinod Kumar Vavilapalli
                Reporter:
                vinodkv Vinod Kumar Vavilapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: