Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12967

NettyRPC races with SparkContext.stop() and throws exception

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.6.0
    • 2.0.0
    • Spark Core
    • None

    Description

      This happens frequently and can be confusing:

      16/01/22 13:07:34 INFO ShutdownHookManager: Shutdown hook called
      16/01/22 13:07:34 WARN Dispatcher: Message RemoteProcessDisconnected(c1706.halxg.cloudera.com:52665) dropped.
      java.lang.IllegalStateException: RpcEnv already stopped.
      at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:165)
      at org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:110)
      at org.apache.spark.rpc.netty.NettyRpcHandler.channelInactive(NettyRpcEnv.scala:618)
      at org.apache.spark.network.server.TransportRequestHandler.channelInactive(TransportRequestHandler.java:99)
      at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:104)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
      at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:257)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
      at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:183)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:208)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:194)
      at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:828)
      at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:621)
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:328)
      at io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:627)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:362)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      at java.lang.Thread.run(Thread.java:745)

      Attachments

        Activity

          People

            nravi Nishkam Ravi
            nravi Nishkam Ravi
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: