[MAPREDUCE-6511] MRAppMaster second attempt starting on the same node as a previously failed MRAppMaster attempt

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.4.1
• Component/s: applicationmaster, scheduler

Environment

EMR

• Assignee: Unassigned
• Reporter: Neil Jonkers
• Created: 12/Oct/15 20:09
• Updated: 28/Oct/15 17:54
• Resolved: 13/Oct/15 08:40

Description

Scenario: MRAppMaster attempt one executed on node that experience hardware issue.

Now the second attempt of the Application Master was scheduled on the same node.
Section from RM log for first APP Master attempt:

2015-10-09 05:54:10,857 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): YARN label is enabled with AM labels CORE
2015-10-09 05:54:10,859 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1444369886652_0001_000001 State change from SUBMITTED to SCHEDULED
2015-10-09 05:54:10,942 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): assignContainers: node=ip-172-31-39-137.us-west-2.compute.internal application=1 priority=0 request=

{Priority: 0, Capability: <memory:15104, vCores:1>, # Containers: 1, Labels: CORE, Location: *, Relax Locality: true}

type=OFF_SWITCH
2015-10-09 05:54:10,973 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl (ResourceManager Event Processor): container_1444369886652_0001_01_000001 Container Transitioned from NEW to ALLOCATED
2015-10-09 05:54:10,973 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger (ResourceManager Event Processor): USER=hadoop OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1444369886652_0001 CONTAINERID=container_1444369886652_0001_01_000001

Section from RM log for second APP Master attempt:

2015-10-09 07:29:10,483 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): YARN label is enabled with AM labels CORE
2015-10-09 07:29:10,483 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl (AsyncDispatcher event handler): appattempt_1444369886652_0001_000002 State change from SUBMITTED to SCHEDULED
2015-10-09 07:29:10,498 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): assignContainers: node=ip-172-31-39-137.us-west-2.compute.internal application=1 priority=0 request=

{Priority: 0, Capability: <memory:15104, vCores:1>, # Containers: 1, Labels: CORE, Location: *, Relax Locality: true}

type=OFF_SWITCH
2015-10-09 07:29:10,499 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl (ResourceManager Event Processor): container_1444369886652_0001_02_000001 Container Transitioned from NEW to ALLOCATED
2015-10-09 07:29:10,499 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger (ResourceManager Event Processor): USER=hadoop OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1444369886652_0001 CONTAINERID=container_1444369886652_0001_02_000001

Issue Links

• is duplicated by: YARN-2005 Blacklisting support for scheduling AMs