Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3148

StormServerPipelineFactory can deserialize messages incorrectly

    Details

      Description

      We recently ran into an integration test failure. (TestingTest).

      It looks like the only way for this error to happen would be if there was an internal bug in kryo, or if we were using Output from multiple threads.  It is the latter.

       

      org.apache.storm.messaging.netty.Server creates a single KryoValuesSerializer and KryoValuesDeserializer instance.  These get passed through StormServerPipelineFactory and added to each channel (if there are multiple channels there are multiple threads) and we can mess up both encoding and decoding messages.

       

      2018-07-12 17:41:49.408 [Netty-server-localhost-1030-worker-1] ERROR org.apache.storm.messaging.netty.StormServerHandler - server errors in handling the request
      org.apache.storm.shade.io.netty.handler.codec.EncoderException: java.lang.ArrayIndexOutOfBoundsException
          at org.apache.storm.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:106) ~[shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1066) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.messaging.netty.Server.channelActive(Server.java:261) [storm-client-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.messaging.netty.StormServerHandler.channelActive(StormServerHandler.java:40) [storm-client-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1422) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:941) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:518) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:465) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [shaded-deps-2.0.0.y.jar:2.0.0.y]
          at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
      Caused by: java.lang.ArrayIndexOutOfBoundsException
          at java.lang.System.arraycopy(Native Method) ~[?:1.8.0_60]
          at com.esotericsoftware.kryo.io.Output.toBytes(Output.java:130) ~[kryo-3.0.3.jar:?]
          at org.apache.storm.serialization.KryoValuesSerializer.serializeObject(KryoValuesSerializer.java:50) ~[storm-client-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.messaging.netty.BackPressureStatus.buffer(BackPressureStatus.java:68) ~[storm-client-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.messaging.netty.BackPressureStatusEncoder.encode(BackPressureStatusEncoder.java:34) ~[storm-client-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.messaging.netty.BackPressureStatusEncoder.encode(BackPressureStatusEncoder.java:24) ~[storm-client-2.0.0.y.jar:2.0.0.y]
          at org.apache.storm.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:88) ~[shaded-deps-2.0.0.y.jar:2.0.0.y]
          ... 27 more
         

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                revans2 Robert Joseph Evans
                Reporter:
                revans2 Robert Joseph Evans
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h 10m
                  3h 10m