Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.10.0
-
None
-
None
-
None
Description
Recently, we encounter an confused issue. The client disconnection warning disappears in system log. However, sometimes, this message appears in system log. There is a cluster consisting of three node. A client sends many creation requests and then read the node created by the first request. The client read operation failed due to missing node. We watch the system log. Sometimes, there is a client disconnection warning. Sometimes, there is not. This incomplete system log mislead client judgement on the problem.
After investigating, when NIOServerCnxn.doIO is stuck in any IO point in this function and the stuck time exceeds 20s, the client disconnection warning will disappear. If the stuck time is less than 20s, the client disconnection warning will appear in system log.
We find that the root cause is that selectorThread is set as the daemon thread. When one node encounter the fail-slow nic, the client disconnects with the node. If the NIOServerCnxn.doIO is stuck and the stuck time exceeds 20s, the corresponding selectorThread will be killed by JVM. Hence, the client disconnection warning is missed.
Attached logs(20s) contain CancelledKeyException, but logs(25) do not contain.
Are there any comments to figure out this issues and improve the diagnosability of ZooKeeper？ I will very appreciate them.