ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-1833 fix windows build
  3. ZOOKEEPER-1838

ZooKeeper shutdown hangs indefinitely at NioServerSocketChannelFactory.releaseExternalResources

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 3.4.6
    • Component/s: server
    • Labels:
      None

      Description

      Zookeeper shutdown hangs when releasing external resources. This has been observed when executing NioNettySuiteTest.

      "main" prio=6 tid=0x01498400 nid=0x2328 waiting on condition [0x0158e000..0x0158fe28]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x22f58918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
      	at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1244)
      	at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:87)
      	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:146)
      	at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:324)
      	at org.apache.zookeeper.server.NettyServerCnxnFactory.shutdown(NettyServerCnxnFactory.java:345)
      	at org.apache.zookeeper.test.ClientBase.shutdownServerInstance(ClientBase.java:355)
      	at org.apache.zookeeper.test.ClientBase.stopServer(ClientBase.java:422)
      

        Issue Links

          Activity

          Rakesh R created issue -
          Rakesh R made changes -
          Field Original Value New Value
          Attachment ZOOKEEPER-1838.patch [ 12618586 ]
          Hide
          Flavio Junqueira added a comment -

          Do you know why bumping the netty version up solves the problem?

          Show
          Flavio Junqueira added a comment - Do you know why bumping the netty version up solves the problem?
          Hide
          Rakesh R added a comment -

          In my windows env, org.apache.zookeeper.test.NioNettySuiteTest test case is hanging indefinitely. When analyzed the threaddump, its waiting to release the external resources. When gone through the netty bug fixes, there are many potential bug fixes happened after 3.2.4 . Also, there is a fix https://issues.jboss.org/browse/NETTY-417 which talks about the possibilities of open channels after closure. After bumping the version, have seen the test case is consistently passing.

          Show
          Rakesh R added a comment - In my windows env, org.apache.zookeeper.test.NioNettySuiteTest test case is hanging indefinitely. When analyzed the threaddump, its waiting to release the external resources. When gone through the netty bug fixes, there are many potential bug fixes happened after 3.2.4 . Also, there is a fix https://issues.jboss.org/browse/NETTY-417 which talks about the possibilities of open channels after closure. After bumping the version, have seen the test case is consistently passing.
          Hide
          Flavio Junqueira added a comment -

          Could you send a message to the dev list and ask if anyone has a problem with us bumping up the netty version? This message will go to the same list, but I don't think folks pay close attention to what's going on in random jiras.

          Show
          Flavio Junqueira added a comment - Could you send a message to the dev list and ask if anyone has a problem with us bumping up the netty version? This message will go to the same list, but I don't think folks pay close attention to what's going on in random jiras.
          Rakesh R made changes -
          Hide
          Rakesh R added a comment -

          Thanks Flavio for the comment. I've sent mail to dev@zookeeper.apache.org list, this will really help to know more about the compatibilty issues and feedback. Also I have attached the threaddump to the jira.

          Show
          Rakesh R added a comment - Thanks Flavio for the comment. I've sent mail to dev@zookeeper.apache.org list, this will really help to know more about the compatibilty issues and feedback. Also I have attached the threaddump to the jira.
          Flavio Junqueira made changes -
          Link This issue is related to ZOOKEEPER-1715 [ ZOOKEEPER-1715 ]
          Hide
          Flavio Junqueira added a comment -

          I had the impression we had talked about upgrading the netty version at some point, so I searched and found it: ZOOKEEPER-1715

          Show
          Flavio Junqueira added a comment - I had the impression we had talked about upgrading the netty version at some point, so I searched and found it: ZOOKEEPER-1715
          Hide
          Michi Mutsuzaki added a comment -

          Ok, so let's backport ZOOKEEPER-1715 to 3.4. I'll close this ticket and reopen ZOOKEEPER-1715.

          Show
          Michi Mutsuzaki added a comment - Ok, so let's backport ZOOKEEPER-1715 to 3.4. I'll close this ticket and reopen ZOOKEEPER-1715 .
          Hide
          Michi Mutsuzaki added a comment -

          This is a dup of ZOOKEEPER-1715.

          Show
          Michi Mutsuzaki added a comment - This is a dup of ZOOKEEPER-1715 .
          Michi Mutsuzaki made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Hide
          Rakesh R added a comment -

          Thanks Flavio, Michi for looking at this and nice to hear about the netty upgradation plans in 3.4 branch.

          Show
          Rakesh R added a comment - Thanks Flavio, Michi for looking at this and nice to hear about the netty upgradation plans in 3.4 branch.
          Hide
          Flavio Junqueira added a comment -

          Closing issues after releasing 3.4.6.

          Show
          Flavio Junqueira added a comment - Closing issues after releasing 3.4.6.
          Flavio Junqueira made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          7h 5m 1 Michi Mutsuzaki 13/Dec/13 19:24
          Resolved Resolved Closed Closed
          89d 22h 52m 1 Flavio Junqueira 13/Mar/14 18:17

            People

            • Assignee:
              Rakesh R
              Reporter:
              Rakesh R
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development