[SLIDER-69] Uber JIRA: Slider apps to withstand failures

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.30
• Component/s: appmaster
• Sprint: Slider August #1, Slider August #2, Slider September #1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 19/May/14 13:35
• Updated: 02/Oct/14 17:13
• Resolved: 02/Oct/14 17:13

Description

Slider-deployed applications should be robust against failure of

1. the App Master: app to keep running, component registration data to remain; restarted AM to bring application back into desired state with minimal container loss/restart
2. the RM: AM and containers to continue
3. Nodes & Racks containing containers: AM to request replacement containers.
4. network partitions. Rely on NMs to kill containers, treat as a rack/node failure.

Issue Links

• depends upon: SLIDER-285 Slider Agents to bind and work with restarted AM

Sub-Tasks

• use a window for tracking container failures: Resolved
• Slider to work on HA NNs: Resolved
• slider registry code to gracefully handle (transient) ZK outages: Resolved
• add integral/configurable chaos monkey to slider AM: Resolved
• Implement scalable failure threshold based on percentage of instances failing over a time period: Resolved
• AM must react to NM failure events by releasing containers: Resolved
• AM to notify providers when restarted: Resolved
• AM to notify containers on managed container release: Resolved
• add scheduled executor to AM and event queue for async action dispatch: Resolved
• failure thresholds to be settable per-role: Resolved
• factor out internal keys into class InternalKeys: Resolved