[SLIDER-82] Support ANTI_AFFINITY_REQUIRED option

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Slider 0.91
• Component/s: appmaster
• Sprint: Slider September #1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 20/May/14 11:30
• Updated: 17/Nov/16 21:36
• Resolved: 15/Jun/16 08:45

Description

slider has an anti-affinity flag in roles (visible in resources.json?), which is ignored.

YARN-1042 promises this for YARN, slider will need

1. flag in resources.json
2. use in container requests

we may also want two policies: anti-affinity-desired, and -required. Then if required nodes get >1 container for the same component type on the same node, it'd have to request a new one and return the old one (Risk: getting the same one back).

Issue Links

• depends upon: SLIDER-945 build up history of request/release/failures for each component and serve in the REST API (and web UI?)
• depends upon: YARN-1412 Allocating Containers on a particular Node in Yarn
• is depended upon by: SLIDER-823 Über-JIRA : placement phase 3
• is duplicated by: SLIDER-726 Different instances of the same component should run on different nodes (anti-affinity)
• is related to: SLIDER-1034 TestAgentAAEcho failed with #of pending instances == 49
• relates to: YARN-1042 add ability to specify affinity/anti-affinity in container requests
• relates to: YARN-5907 [Umbrella] [YARN-1042] add ability to specify affinity/anti-affinity in container requests
• relates to: TWILL-87 Adding Container Placement control (Placement Policy API)

Sub-Tasks

• build node map from yarn update reports; serve via REST/IPC: Resolved
• Write mock/unit tests for AA placement: Resolved
• RoleHistory to (carefully) share RoleStatus instances with AppState: Resolved
• Implement sequential AA assignment without location constraints: Resolved
• Use nodemap to build up location restrictions on AA placement: Resolved
• Add mock test to simulate AA failure and restart of AA request sequence: Resolved
• Add functional tests of AA placement: Resolved
• revisit why OutstandingRequest.buildContainerRequest() sets label==null on a placed request: Resolved
• AM web UI to show state of AA request: Resolved
• add minicluster test of AA placement: Resolved
• add mock test of failure of AA container and re-request; fix any failures: Resolved
• Mock AA test for nodemap not updated: Resolved
• add "nodemap" command to get the (JSON) nodemap of the YARN cluster: Resolved
• Update slider docs with coverage of AA placement: Resolved
• AM can't make YarnClient calls on a secure cluster: Resolved
• review label logic in AA code: Resolved
• add regression test to verify latest role history format reload: Resolved
• rename NO_DATA_LOCALITY placement policy to ANYWHERE: Resolved