[SLIDER-758] Slider placement requests to skip unreliable nodes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Slider 0.60
• Fix Version/s: Slider 0.70
• Component/s: appmaster
• Sprint: Slider Jan #2
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/Jan/15 14:16
• Updated: 28/Jan/15 18:34
• Resolved: 28/Jan/15 18:34

Description

As discussed on the developer list; slider's "prefer previously used nodes" is biased towards recently used nodes —even when those nodes are failing to successfully launch containers.

As we already track node failure rates, the placement logic can be enhanced to not generate "placed" requests on nodes with a (recent) failure history of that component type.

The initial iteration of this feature will not use the YARN blacklisting APIs, instead build up history in the AM, history that will be lost on AM restart. Accordingly, even unplaced requests may end being scheduled on the unreliable nodes.

This strategy (which we could revisit in future), combined with a regular reset of the failure counters, stops slider blacklisting nodes whose failure rate was high some time previously —but which is now reliable again.

Testing: primarily via mocking

Issue Links

• duplicates: SLIDER-743 Include node failure history when choosing placement hints