Blocker Description
We have two solr cloud nodes setup on the Zookeeper.
SOLR version is 5.2.1 and Zookeeper Version is 3.4.6.
SOLR CLOUD Architecture has Listing which is connected to shards which in turn is connected to 2 SOLR CLOUD NODES.
Our Production API which brings listing from SOLR nodes became unavailable.SOLR nodes logs were not generated and did not log any activity for 6 hours when Production was unavailable.SOLR files got rotated with the timestamp when we restarted SOLR service and Zookeeper.out showed below errors in log at the start of downtime:
EndOfStreamException: Unable to read additional data from client sessionid 0x156a6ca33180002, likely client has closed socket
2016-09-26 04:25:07,545 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x156a6ca33180002, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
ZOO.CFG has below settings:
[Replaced IP's Intentionally with x's for security purpose]
server.1=172.x.x.x:2888:3888
server.2=172.x.x.x:2888:3888
server.3=172.x.x.x:2888:3888
Please let us know what could be the probable cause that our API's were not able to connect to SOLR nodes completely which resulted in Production downtime.
Thanks
Pahuni