Details
Trivial Description
I'm running a Spark program (code attached) in Spark local mode using AsyncKuduClient that is attempting to do the following:
1) read timeseries data from Kafka
2) scan Kudu in parallel to see if there is an existing entry in Kudu for tags corresponding to the data
3) insert it if not (using a common session so that we don't overload the server with RPCs)
I reliably get the following error on the client albeit with different data and rpcids and variance in how long it takes, although it's always within the first minute or so:
15/10/19 01:07:01 ERROR TabletClient: [Peer 290d9768dc204615afb4d1d48c568c92] Invalid rpcid: 1 found in ReplayingDecoderBuffer(ridx=9, widx=19)=
[0, 0, 0, 15, 4, 8, 1, 16, 0, 9, 25, 0, 48, 127, 107, 52, -84, 38, 82]
15/10/19 01:07:01 ERROR TabletClient: [Peer 290d9768dc204615afb4d1d48c568c92] Unexpected exception from downstream on [id: 0x7f93c7a9, /10.20.9.
170:34716 :> /10.20.9.170:7050]
org.kududb.client.NonRecoverableException: [Peer 290d9768dc204615afb4d1d48c568c92] Invalid rpcid: 1 found in ReplayingDecoderBuffer(ridx=9, widx
=19)=[0, 0, 0, 15, 4, 8, 1, 16, 0, 9, 25, 0, 48, 127, 107, 52, -84, 38, 82]
at org.kududb.client.TabletClient.decode(TabletClient.java:359)
at org.kududb.client.TabletClient.decode(TabletClient.java:83)
at org.kududb.client.shaded.org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at org.kududb.client.shaded.org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.kududb.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.kududb.client.TabletClient.handleUpstream(TabletClient.java:593)
at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipel
ine.java:791)
at org.kududb.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
at org.kududb.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.kududb.client.AsyncKuduClient$TabletClientPipeline.sendUpstream(AsyncKuduClient.java:1618)
at org.kududb.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.kududb.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.kududb.client.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.kududb.client.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
I don't see any errors on the master server or tablet server (running both on localhost with logtostderr but no other logging arguments).
Any ideas? Program attached. Happy to answer any questions or get other logs.