[KUDU-1669] Java ITClient test can orphan processes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: client
• Code Review: https://gerrit.cloudera.org/#/c/4598/
• Assignee: Jean-Daniel Cryans
• Reporter: Adar Dembo
• Created: 30/Sep/16 21:51
• Updated: 05/Oct/16 23:46
• Resolved: 05/Oct/16 23:46

Description

Saw this with ITClient but it can happen to any test that uses a MiniCluster, restarts a process, and interrupts its own threads. Maybe ITClient is the only example of that.

Here's the interesting stack trace:

21:18:12.316 [ERROR - Thread-8] (ITClient.java:134) Couldn't restart a master
java.lang.InterruptedException: sleep interrupted
        at java.lang.Thread.sleep(Native Method)
        at org.apache.kudu.client.MiniKuduCluster.configureAndStartProcess(MiniKuduCluster.java:239)
        at org.apache.kudu.client.MiniKuduCluster.restartDeadProcessOnPort(MiniKuduCluster.java:282)
        at org.apache.kudu.client.MiniKuduCluster.restartDeadMasterOnPort(MiniKuduCluster.java:256)
        at org.apache.kudu.client.BaseKuduTest.restartLeaderMaster(BaseKuduTest.java:431)
        at org.apache.kudu.client.ITClient$ChaosThread.restartMaster(ITClient.java:222)
        at org.apache.kudu.client.ITClient$ChaosThread.run(ITClient.java:161)
        at java.lang.Thread.run(Thread.java:745)

Because of this thread interruption, the newly restarted process never makes it into BaseKuduTest.masterProcesses, which means it isn't destroyed when the test class cleans up. All sorts of bad stuff can happen then. It's possible the process is completely orphaned on the test machine (though I imagine we'd kill it eventually). I noticed this because in one of my precommit test runs ITClient left behind a test directory; presumably because the orphaned master continued to write files even after the test directory was cleaned up.