[YARN-6050] AMs can't be scheduled on racks or nodes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.0, 3.0.0-alpha2
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Target Version/s: 2.9.0, 3.0.0-alpha4
• Hadoop Flags: Reviewed
• Assignee: Robert Kanter
• Reporter: Robert Kanter
• Created: 03/Jan/17 18:23
• Updated: 25/Oct/19 20:25
• Resolved: 28/Mar/17 21:33

Description

Yarn itself supports rack/node aware scheduling for AMs; however, there currently are two problems:

1. To specify hard or soft rack/node requests, you have to specify more than one ResourceRequest. For example, if you want to schedule an AM only on "rackA", you have to create two ResourceRequest, like this:

   ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
   ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS, true);
   

   The problem is that the Yarn API doesn't actually allow you to specify more than one ResourceRequest in the ApplicationSubmissionContext. The current behavior is to either build one from getResource or directly from getAMContainerResourceRequest, depending on if getAMContainerResourceRequest is null or not. We'll need to add a third method, say getAMContainerResourceRequests, which takes a list of ResourceRequest so that clients can specify the multiple resource requests.

2. There are some places where things are hardcoded to overwrite what the client specifies. These are pretty straightforward to fix.

Issue Links

• relates to: MAPREDUCE-6871 Allow users to specify racks and nodes for strict locality for AMs
• relates to: OOZIE-2874 Make the Launcher Mapper map-only job's InputFormat class pluggable
• links to: ReviewBoard

Sub-Tasks

• Create CS test for YARN-6050: Open