[SOLR-9170] Need detailed diagnostics for Zookeeper Connection Issues

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 6.0.1
• Component/s: SolrCloud

Environment

Window or Linux

• Assignee: Unassigned
• Reporter: harcor
• Created: 28/May/16 15:14
• Updated: 28/May/16 18:13

Description

When Solr disconnects from Zookeeper or Zookeeper disconnects from for some abnormal reason it is difficult to identify the root of the problem. If we could have a mode that provide additional diagnostics both in the solr log and zookeeper log.

IE: Solr you could pull system stats about the current state of solr. What is the load / memory.

IE2: For zookeeper, if might show that the solr socket was abnormally closed but gives no further info. See the example below.

2016-05-27 00:59:53,357 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x154a2e5fd9b0b2a with negotiated timeout 120000 for client /17.34.240.66:356202016-05-27 01:00:57,627 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x154a2e5fd9b0b24, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745)2016-05-27 01:00:57,627 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /17.34.240.66:35266 which had sessionid 0x154a2e5fd9b0b242016-05-27 01:01:00,873 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /17.34.240.66:35508 which had sessionid 0x154a2e5fd9b0b292016-05-27 01:03:06,425 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /17.34.240.66:35620 which had sessionid 0x154a2e5fd9b0b2a