[ZOOKEEPER-2044] CancelledKeyException in zookeeper branch-3.4

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.6
• Fix Version/s: 3.4.10
• Component/s: server
• Labels: pull-request-available

Environment

Red Hat Enterprise Linux Server release 6.2

• Assignee: Michael Han
• Reporter: shamjith antholi
• Created: 22/Sep/14 10:17
• Updated: 27/Mar/20 13:10
• Resolved: 31/Jan/17 16:40

Description

I am getting cancelled key exception in zookeeper (version 3.4.5). Please see the log below. When this error is thrown, the connected solr shard is going down by giving the error "Failed to index metadata in Solr,StackTrace=SolrError: HTTP status 503.Reason: {"responseHeader":

{"status":503,"QTime":204}

,"error":{"msg":"ClusterState says we are the leader, but locally we don't think so","code":503" and ultimately the current activity is going down. Could you please give a solution for this ?

Zookeper log
----------------------------------------------------------
2014-09-16 02:58:47,799 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] - Client attempting to renew session 0x24868e7ca980003 at /172.22.0.5:58587
2014-09-16 02:58:47,800 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@107] - Revalidating client: 0x24868e7ca980003
2014-09-16 02:58:47,802 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@588] - Invalid session 0x24868e7ca980003 for client /172.22.0.5:58587, probably expired
2014-09-16 02:58:47,803 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /172.22.0.5:58587 which had sessionid 0x24868e7ca980003
2014-09-16 02:58:47,810 [myid:1] - ERROR [CommitProcessor:1:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1113)
at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1327)
at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:384)
at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:304)
at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)

Issue Links

• is duplicated by: ZOOKEEPER-1237 ERRORs being logged when queued responses are sent after socket has closed.
• is related to: ZOOKEEPER-2677 Verify the occurrence of CancelledKeyException in zookeeper branch-3.5 and above
• links to: GitHub Pull Request #156
• links to: GitHub Pull Request #156