Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1221

Invalid Rpcid using Java client

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Trivial
    • Resolution: Unresolved
    • Affects Version/s: Public beta
    • Fix Version/s: None
    • Component/s: client
    • Labels:
      None
    • Environment:
      Java client built from kudu git hash 1d8f3b6f98f1bf564a18214d60dacdd02e3d5573

      Description

      I'm running a Spark program (code attached) in Spark local mode using AsyncKuduClient that is attempting to do the following:
      1) read timeseries data from Kafka
      2) scan Kudu in parallel to see if there is an existing entry in Kudu for tags corresponding to the data
      3) insert it if not (using a common session so that we don't overload the server with RPCs)

      I reliably get the following error on the client albeit with different data and rpcids and variance in how long it takes, although it's always within the first minute or so:

      15/10/19 01:07:01 ERROR TabletClient: [Peer 290d9768dc204615afb4d1d48c568c92] Invalid rpcid: 1 found in ReplayingDecoderBuffer(ridx=9, widx=19)=
      [0, 0, 0, 15, 4, 8, 1, 16, 0, 9, 25, 0, 48, 127, 107, 52, -84, 38, 82]
      15/10/19 01:07:01 ERROR TabletClient: [Peer 290d9768dc204615afb4d1d48c568c92] Unexpected exception from downstream on [id: 0x7f93c7a9, /10.20.9.
      170:34716 :> /10.20.9.170:7050]
      org.kududb.client.NonRecoverableException: [Peer 290d9768dc204615afb4d1d48c568c92] Invalid rpcid: 1 found in ReplayingDecoderBuffer(ridx=9, widx
      =19)=[0, 0, 0, 15, 4, 8, 1, 16, 0, 9, 25, 0, 48, 127, 107, 52, -84, 38, 82]
      at org.kududb.client.TabletClient.decode(TabletClient.java:359)
      at org.kududb.client.TabletClient.decode(TabletClient.java:83)
      at org.kududb.client.shaded.org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
      at org.kududb.client.shaded.org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
      at org.kududb.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      at org.kududb.client.TabletClient.handleUpstream(TabletClient.java:593)
      at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipel
      ine.java:791)
      at org.kududb.client.shaded.org.jboss.netty.handler.timeout.ReadTimeoutHandler.messageReceived(ReadTimeoutHandler.java:184)
      at org.kududb.client.shaded.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.kududb.client.shaded.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.kududb.client.AsyncKuduClient$TabletClientPipeline.sendUpstream(AsyncKuduClient.java:1618)
      at org.kududb.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.kududb.client.shaded.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
      at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
      at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
      at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
      at org.kududb.client.shaded.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
      at org.kududb.client.shaded.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      at org.kududb.client.shaded.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

      I don't see any errors on the master server or tablet server (running both on localhost with logtostderr but no other logging arguments).

      Any ideas? Program attached. Happy to answer any questions or get other logs.

        Attachments

        1. KuduRunner.scala
          7 kB
          Fred Wulff

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              frew Fred Wulff
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: