Uploaded image for project: 'Livy'
  1. Livy
  2. LIVY-188

table magic crash on large dataset included unicode string

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.2, 0.3
    • Fix Version/s: 0.3
    • Component/s: Interpreter, RSC
    • Labels:
      None

      Description

      in python notebook, using table magic with large dataset include unicode string.
      return Exception.

      Test Code

      test_map = []
      for x in range(0, 100, 1) :
          test_list = []
          for y in range(0, 100, 1) :
              test_list.append(x)
              test_list.append(y)
              test_list.append(u"\u263A")
          test_map.append(test_list)
      
      %table test_map
      

      Livy Server Exception

      io.netty.handler.codec.DecoderException: com.cloudera.livy.shaded.kryo.kryo.KryoException: Buffer underflow.
              at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
              at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
              at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:108)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
              at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
              at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
              at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: com.cloudera.livy.shaded.kryo.kryo.KryoException: Buffer underflow.
              at com.cloudera.livy.shaded.kryo.kryo.io.Input.require(Input.java:164)
              at com.cloudera.livy.shaded.kryo.kryo.io.Input.readUtf8_slow(Input.java:561)
              at com.cloudera.livy.shaded.kryo.kryo.io.Input.readUtf8(Input.java:535)
              at com.cloudera.livy.shaded.kryo.kryo.io.Input.readString(Input.java:465)
              at com.cloudera.livy.shaded.kryo.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:171)
              at com.cloudera.livy.shaded.kryo.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:160)
              at com.cloudera.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:776)
              at com.cloudera.livy.client.common.Serializer.deserialize(Serializer.java:63)
              at com.cloudera.livy.rsc.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:77)
              at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42)
              at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
      

      tested By master branch, spark 1.6.1

        Attachments

          Activity

            People

            • Assignee:
              purechoc hyunwoo cho
              Reporter:
              purechoc hyunwoo cho
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: