[HBASE-24658] Update PolicyBasedChaosMonkey to handle uncaught exceptions

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.6
• Component/s: integration tests
• Assignee: Nick Dimiduk
• Reporter: Nick Dimiduk
• Created: 29/Jun/20 23:32
• Updated: 26/Aug/20 05:50
• Resolved: 21/Jul/20 17:03

Description

Running ServerKillingChaosMonkey via RESTApiClusterManager for any duration of time slowly leaks region servers. I see failures on the RESTApi side go unreported on the ChaosMonkey side. It seems like RuntimeExceptions are being thrown and lost.

PolicyBasedChaosMonkey uses a primitive means of thread management anyway. Update to use a thread pool, thread groups, and an uncaughtExceptionHandler.

Issue Links

• links to: GitHub Pull Request #2000
• links to: GitHub Pull Request #2105
• links to: GitHub Pull Request #2106
• links to: GitHub Pull Request #2107
• links to: GitHub Pull Request #2108