[SLIDER-202] add integral/configurable chaos monkey to slider AM

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.40
• Component/s: appmaster, test
• Sprint: Slider August #1, Slider August #2, Slider September #1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 02/Jul/14 12:51
• Updated: 15/Aug/14 11:05
• Resolved: 13/Aug/14 17:18

Description

The slider funtests trigger container and AM failure in the test code via RPC calls "killContainer" and "amSuicide". This is good for standalone tests, but complicates functional test deployments where we can't easily run such logic client-side.

Proposed: a built in chaos monkey service in the AM which, when enabled, will at given intervals check to see if a container or AM restart should be triggered based on some preconfigured probabilities.

This will be embedded and can be used in production if desired.
e.g, short test:

interval = 60s, P(AM)=0.1, P(Container)=1.0

long test

interval = 600s, P(AM)=0.01, P(Container)=0.5

production

interval = 24h, P(AM)=0., P(Container)=0.1

Issue Links

• breaks: SLIDER-325 codahale metrics not being uploaded to AM classpath