[LIVY-188] table magic crash on large dataset included unicode string

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.2, 0.3
• Fix Version/s: 0.3
• Component/s: Interpreter, RSC
• Assignee: hyunwoo cho
• Reporter: hyunwoo cho
• Created: 07/Jul/16 07:47
• Updated: 13/Jul/16 22:20
• Resolved: 13/Jul/16 22:20

Description

in python notebook, using table magic with large dataset include unicode string.
return Exception.

Test Code

test_map = []
for x in range(0, 100, 1) :
    test_list = []
    for y in range(0, 100, 1) :
        test_list.append(x)
        test_list.append(y)
        test_list.append(u"\u263A")
    test_map.append(test_list)

%table test_map

Livy Server Exception

io.netty.handler.codec.DecoderException: com.cloudera.livy.shaded.kryo.kryo.KryoException: Buffer underflow.
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
        at io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:108)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.cloudera.livy.shaded.kryo.kryo.KryoException: Buffer underflow.
        at com.cloudera.livy.shaded.kryo.kryo.io.Input.require(Input.java:164)
        at com.cloudera.livy.shaded.kryo.kryo.io.Input.readUtf8_slow(Input.java:561)
        at com.cloudera.livy.shaded.kryo.kryo.io.Input.readUtf8(Input.java:535)
        at com.cloudera.livy.shaded.kryo.kryo.io.Input.readString(Input.java:465)
        at com.cloudera.livy.shaded.kryo.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:171)
        at com.cloudera.livy.shaded.kryo.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:160)
        at com.cloudera.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:776)
        at com.cloudera.livy.client.common.Serializer.deserialize(Serializer.java:63)
        at com.cloudera.livy.rsc.rpc.KryoMessageCodec.decode(KryoMessageCodec.java:77)
        at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)

tested By master branch, spark 1.6.1