[YARN-5907] [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha1
• Component/s: resourcemanager
• Assignee: Wangda Tan
• Reporter: Steve Loughran
• Created: 17/Nov/16 21:36
• Updated: 31/Jan/18 23:37

Description

container requests to the AM should be able to request anti-affinity to ensure that things like Region Servers don't come up on the same failure zones.

Similarly, you may be able to want to specify affinity to same host or rack without specifying which specific host/rack. Example: bringing up a small giraph cluster in a large YARN cluster would benefit from having the processes in the same rack purely for bandwidth reasons.

This JIRA is cloned umbrella JIRA of YARN-1042, discussions / designs / POC patches, etc. please refer to YARN-1042.

Issue Links

• depends upon: YARN-624 Support gang scheduling in the AM RM protocol
• is a clone of: YARN-1042 add ability to specify affinity/anti-affinity in container requests
• is depended upon by: YARN-896 Roll up for long-lived services in YARN
• is depended upon by: SAMZA-74 Use anti-affinity in YARN container requests
• is duplicated by: YARN-2969 allocate resource on different nodes for task
• is related to: SLIDER-82 Support ANTI_AFFINITY_REQUIRED option
• relates to: YARN-2027 YARN ignores host-specific resource requests
• relates to: TWILL-87 Adding Container Placement control (Placement Policy API)
• relates to: YARN-4879 Enhance Allocate Protocol to Identify Requests Explicitly
• requires: YARN-5139 [Umbrella] Move YARN scheduler towards global scheduler

Sub-Tasks

• Add affinity/anti-affinity field to ResourceRequest API: Resolved