[ZOOKEEPER-2838] race-condition when shutting down NIOServerCnxnFactory yields CancelledKeyException

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.6, 3.4.7, 3.4.8, 3.4.9, 3.4.10, 3.5.0, 3.5.1, 3.5.2, 3.5.3
• Component/s: server
• Assignee: Unassigned
• Reporter: Paul Millar
• Created: 05/Jul/17 08:18
• Updated: 05/Jul/17 08:18

Description

The problem stems from closing the ServerSocketChannel before stopping the thread(s) working with the corresponding Selector. Closing the ServerSocketChannel will invalidate any SelectionKey objects that have been declared. This is equivalent to calling cancel on the SelectionKey. Therefore, after the ServerSocketChannel's close method is called, it is possible that any thread working with a SelectionKey will experience CancelledKeyException.

I noticed the problem with ZooKeeper v3.4.6, which resulted in the following stack-trace:

04 Jul 2017 15:54:15 (zookeeper) [] Ignoring unexpected runtime exception
java.nio.channels.CancelledKeyException: null
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) ~[na:1.8.0_131]
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87) ~[na:1.8.0_131]
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:187) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

From manually inspecting the source code, I see the problem is present in all currently released versions of ZooKeeper.