[KUDU-2687] ITClient retries are broken

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.10.0
• Component/s: java, test
• Code Review: https://gerrit.cloudera.org/c/12820/
• Assignee: Adar Dembo
• Reporter: Adar Dembo
• Created: 02/Feb/19 20:09
• Updated: 22/Mar/19 00:12
• Resolved: 22/Mar/19 00:12

Description

I thought I fixed this with this commit but apparently there's still some other issue at play.

From the most recent failure:

23:58:18.980 [ERROR - Thread-18] (ITClient.java:153) Got error while inserting row 0
org.apache.kudu.client.NonRecoverableException: 
	at org.apache.kudu.client.KuduException.transformException(KuduException.java:132)
	at org.apache.kudu.client.KuduSession.apply(KuduSession.java:93)
	at org.apache.kudu.client.ITClient$WriterThread.run(ITClient.java:264)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
	at org.apache.kudu.client.Connection.enqueueMessage(Connection.java:546)
	at org.apache.kudu.client.RpcProxy.sendRpc(RpcProxy.java:126)
	at org.apache.kudu.client.AsyncKuduClient.sendRpcToTablet(AsyncKuduClient.java:1229)
	at org.apache.kudu.client.AsyncKuduClient$RetryRpcCB.call(AsyncKuduClient.java:1278)
	at org.apache.kudu.client.AsyncKuduClient$RetryRpcCB.call(AsyncKuduClient.java:1268)
	at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
	at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
	at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
	at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:247)
	at org.apache.kudu.client.KuduRpc.callback(KuduRpc.java:294)
	at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:269)
	at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:59)
	at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:131)
	at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:127)
	at org.apache.kudu.client.Connection.messageReceived(Connection.java:391)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.apache.kudu.client.Connection.handleUpstream(Connection.java:243)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more

I'm attaching the full log.