[HBASE-14625] Chaos Monkey should shut down faster

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 2.0.0
• Fix Version/s: 1.2.0, 1.3.0, 2.0.0
• Component/s: integration tests, test
• Assignee: Elliott Neil Clark
• Reporter: Elliott Neil Clark
• Created: 15/Oct/15 23:11
• Updated: 17/Oct/15 02:35
• Resolved: 16/Oct/15 22:25

Description

Right now we have a couple of tests clusters that are just cycling through IT tests. There's a pretty sizable gap between the last MR job stopping and the next one starting. It's almost always waiting on an action to finish.

"main" #1 prio=5 os_prio=0 tid=0x00007f455000d800 nid=0x2a2773 in Object.wait() [0x00007f4556e42000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1245)
	- locked <0x00000003ef054138> (a java.lang.Thread)
	at java.lang.Thread.join(Thread.java:1319)
	at org.apache.hadoop.hbase.chaos.monkies.PolicyBasedChaosMonkey.waitForStop(PolicyBasedChaosMonkey.java:149)
	at org.apache.hadoop.hbase.IntegrationTestBase.cleanUpMonkey(IntegrationTestBase.java:173)
	at org.apache.hadoop.hbase.IntegrationTestBase.cleanUpMonkey(IntegrationTestBase.java:167)
	at org.apache.hadoop.hbase.IntegrationTestBase.cleanUp(IntegrationTestBase.java:139)
	at org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:125)
	at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList.main(IntegrationTestBigLinkedList.java:1686)