[SAMZA-74] Use anti-affinity in YARN container requests

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.6.0
• Component/s: yarn
• Assignee: Unassigned
• Reporter: Chris Riccomini
• Created: 31/Oct/13 22:25
• Updated: 17/Nov/16 21:36

Description

Samza's YARN containers are currently allocated according to whatever default strategy the YARN RM uses. Sometimes, we get all of our containers allocated to a single NM, even when many NMs have available resources.

YARN is in the process of getting an anti-affinity feature (YARN-1042). When this is implemented, we should consider using it to prevent all the containers from ending up on a single host.

NOTE: This anti-affinity feature might conflict with a colocate-processor-with-data feature, if we were trying to colocate stream processors near their data.

Issue Links

• depends upon: YARN-1042 add ability to specify affinity/anti-affinity in container requests
• depends upon: YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests
• relates to: SAMZA-124 Provide AM based antiaffinity to avoid clumping of containers