[MAPREDUCE-4893] MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.3-alpha, 0.23.7
• Component/s: applicationmaster
• Hadoop Flags: Reviewed
• Assignee: Bikas Saha
• Reporter: Bikas Saha
• Created: 19/Dec/12 20:02
• Updated: 13/Nov/13 15:40
• Resolved: 31/Jan/13 00:30

Description

Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3. There are 10 node n1-n10 in the same rack. The RM can give it allocated containers in the list order n5, n2, n1. The way AM map->container assignment happens, the AM will try to assign node local maps to n5, failing which it will assign rack local maps to n5. These rack local maps could be node local on n2 and n1 and would have been assigned to containers on n1 and n2 if the AM had not made an early rack local match for them on n5. This can lead to poor locality.

Issue Links

• is duplicated by: MAPREDUCE-4982 AM hung with one pending map task
• is related to: MAPREDUCE-5622 MRAppMaster doesn't assign all allocated NODE_LOCAL containers to node-local maps
• is related to: MAPREDUCE-5002 AM could potentially allocate a reduce container to a map attempt