Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12967

NettyRPC races with SparkContext.stop() and throws exception

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 2.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      This happens frequently and can be confusing:

      16/01/22 13:07:34 INFO ShutdownHookManager: Shutdown hook called
      16/01/22 13:07:34 WARN Dispatcher: Message RemoteProcessDisconnected(c1706.halxg.cloudera.com:52665) dropped.
      java.lang.IllegalStateException: RpcEnv already stopped.
      at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:165)
      at org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:110)
      at org.apache.spark.rpc.netty.NettyRpcHandler.channelInactive(NettyRpcEnv.scala:618)
      at org.apache.spark.network.server.TransportRequestHandler.channelInactive(TransportRequestHandler.java:99)
      at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:104)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
      at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:257)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
      at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:183)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
      at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:328)
      at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:627)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:362)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      at java.lang.Thread.run(Thread.java:745)

        Attachments

          Activity

            People

            • Assignee:
              nravi Nishkam Ravi
              Reporter:
              nravi Nishkam Ravi
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: