[KUDU-1526] ITClient test is flaky and fails on rerun

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.1
• Fix Version/s: NA
• Component/s: client
• Target Version/s: 1.2.0
• Assignee: Jean-Daniel Cryans
• Reporter: Adar Dembo
• Created: 12/Jul/16 01:17
• Updated: 15/Nov/16 02:59
• Resolved: 15/Nov/16 02:59

Description

00:38:50 Running org.kududb.client.ITClient
00:39:14 Tests run: 4, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 13.604 sec <<< FAILURE! - in org.kududb.client.ITClient
00:39:14 test(org.kududb.client.ITClient)  Time elapsed: 3.915 sec  <<< FAILURE!
00:39:14 java.lang.AssertionError: Look for the last ERROR line in the log that comes from ITCLient
00:39:14 	at org.junit.Assert.fail(Assert.java:88)
00:39:14 	at org.junit.Assert.assertTrue(Assert.java:41)
00:39:14 	at org.junit.Assert.assertFalse(Assert.java:64)
00:39:14 	at org.kududb.client.ITClient.test(ITClient.java:106)
00:39:14 
00:39:14 test(org.kududb.client.ITClient)  Time elapsed: 0.002 sec  <<< FAILURE!
00:39:14 java.lang.AssertionError: Look for the last ERROR line in the log that comes from ITCLient
00:39:14 	at org.junit.Assert.fail(Assert.java:88)
00:39:14 	at org.junit.Assert.assertTrue(Assert.java:41)
00:39:14 	at org.junit.Assert.assertFalse(Assert.java:64)
00:39:14 	at org.kududb.client.ITClient.test(ITClient.java:106)
00:39:14 
00:39:14 test(org.kududb.client.ITClient)  Time elapsed: 0.002 sec  <<< FAILURE!
00:39:14 java.lang.AssertionError: Look for the last ERROR line in the log that comes from ITCLient
00:39:14 	at org.junit.Assert.fail(Assert.java:88)
00:39:14 	at org.junit.Assert.assertTrue(Assert.java:41)
00:39:14 	at org.junit.Assert.assertFalse(Assert.java:64)
00:39:14 	at org.kududb.client.ITClient.test(ITClient.java:106)
00:39:14 
00:39:14 test(org.kududb.client.ITClient)  Time elapsed: 0.002 sec  <<< FAILURE!
00:39:14 java.lang.AssertionError: Look for the last ERROR line in the log that comes from ITCLient
00:39:14 	at org.junit.Assert.fail(Assert.java:88)
00:39:14 	at org.junit.Assert.assertTrue(Assert.java:41)
00:39:14 	at org.junit.Assert.assertFalse(Assert.java:64)
00:39:14 	at org.kududb.client.ITClient.test(ITClient.java:106)
00:39:14 
00:39:14 
00:39:14 Results :
00:39:14 
00:39:14 Failed tests: 
00:39:14 org.kududb.client.ITClient.test(org.kududb.client.ITClient)
00:39:14   Run 1: ITClient.test:106 Look for the last ERROR line in the log that comes from ITCLient
00:39:14   Run 2: ITClient.test:106 Look for the last ERROR line in the log that comes from ITCLient
00:39:14   Run 3: ITClient.test:106 Look for the last ERROR line in the log that comes from ITCLient
00:39:14   Run 4: ITClient.test:106 Look for the last ERROR line in the log that comes from ITCLient

The first error is:

00:38:56.544 [ERROR - Thread-8] (ITClient.java:128) Row count regressed: 499 < 500

The reruns are a mess. JD and I looked at it a bit but aren't sure what's going on. One possibility is that we're not giving the cluster enough time to start back up on rerun. Or maybe the threads launched by ITClient stick around across reruns (they aren't explicitly made into daemon threads).