[LIVY-453] Livy Exception while creating multiple sessions in parallel

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.3
• Fix Version/s: 0.9.0
• Component/s: RSC
• Assignee: Unassigned
• Reporter: Abhishek Rao
• Created: 21/Mar/18 06:21
• Updated: 12/Nov/22 22:09

Description

We're trying to create multiple livy sessions in parallel and then using them. But when we try to create the sessions continuously, we're seeing that few sessions are entering to dead state. We see the below exception in the logs.

 

18/02/27 10:30:20 WARN RSCClient: Client RPC channel closed unexpectedly.

18/02/27 10:30:20 WARN RSCClient: Error stopping RPC.

io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@7a828ea3(uncancellable)

               at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)

               at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)

               at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)

               at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)

               at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)

               at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218)

               at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)

               at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)

               at com.cloudera.livy.rsc.rpc.Rpc.close(Rpc.java:307)

               at com.cloudera.livy.rsc.RSCClient.stop(RSCClient.java:225)

               at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:122)

               at com.cloudera.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:116)

               at com.cloudera.livy.rsc.Utils$2.operationComplete(Utils.java:108)

               at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)

               at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)

               at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)

               at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)

               at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956)

               at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608)

               at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586)

               at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)

               at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)

               at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)

               at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)

               at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)

               at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)

               at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)

               at java.lang.Thread.run(Thread.java:748)

18/02/27 10:30:20 DEBUG RSCClient: Disconnected from context dad7c668-3c09-4ad2-9810-28f684c5ec49, shutdown = false.

 

However, when we create the sessions one after the other (Create session 1 after session 0 is in Idle state), it works fine.