[SOLR-13793] HTTPSolrCall makes cascading calls even when all replicas are down for a collection

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 6.6, 9.0
• Fix Version/s: 7.7.3, 8.3
• Component/s: SolrCloud
• Assignee: Ishan Chattopadhyaya
• Reporter: Kesharee Nandan Vishwakarma
• Created: 25/Sep/19 19:27
• Updated: 07/Jun/20 12:29
• Resolved: 15/Oct/19 09:16

Description

REMOTEQUERY action in HTTPSolrCall ends up making too many cascading remoteQuery calls when all all the replicas of a collection are in down state.
This results in increase in thread count, unresponsive solr nodes and eventually node (one's which have this collection) going out of live nodes.

Example scenario: Consider a cluster with 3 nodes(solr1, solrw1, solr-overseer1). A collection is present on solr1, solrw1 but both replicas are in down state. When a search request is made to solr-overseer1, since replica is not present locally a remote query is made to solr1 (we also consider inactive slices/coreUrls), solr1 also doesn't see an active replica present locally, it forwards to solrw1, again solrw1 will forward request to solr1. This goes on till both of solr1, solrw1 become unresponsive. Attached logs for this.

This is happening because we are considering inactive slices, inactive coreUrl while forwarding requests to nodes.

Steps to reproduce:

1. Bring down all replicas of a collection but ensure nodes containing them are up
2. Make any search call to any of solr nodes for this collection.

Possible fixes:

1. Ensure we select only active slices/coreUrls before making remote queries
2. Put a limit on cascading calls probably limit to number of replicas

 
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:14.458 ERROR (qtp762152757-8772) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:14.451 ERROR (qtp762152757-8774) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:14.333 ERROR (qtp762152757-8771) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:12.661 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
solrw1_1         | 2019-09-24 09:35:12.284 ERROR (qtp762152757-8767) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:12.275 ERROR (qtp762152757-8768) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:12.274 ERROR (qtp762152757-8766) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:12.054 ERROR (qtp762152757-8764) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:12.049 ERROR (qtp762152757-8763) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:12.044 ERROR (qtp762152757-8762) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:11.826 ERROR (qtp762152757-556) [   ] o.a.s.s.HttpSolrCall null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
solrw1_1         | 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
solrw1_1         | 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
solrw1_1         | 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
solrw1_1         | 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
solrw1_1         | 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
solrw1_1         | 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
solrw1_1         | 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
solrw1_1         | 	at org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:82)
solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$20(CollectionsHandler.java:664)
solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:922)
solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:228)
solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215)
solrw1_1         | 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:749)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:730)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         |
zook1_1          | 2019-09-24 09:35:30,491 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16d6294ad750020 type:delete cxid:0xd zxid:0x9a5 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/election/102846757204197386-solrw1:8983_solr-n_0000000036 Error:KeeperErrorCode = NoNode for /solr/overseer_elect/election/102846757204197386-solrw1:8983_solr-n_0000000036
zook1_1          | 2019-09-24 09:35:30,527 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16d6294ad750020 type:create cxid:0x12 zxid:0x9a7 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
solrw1_1         | 2019-09-24 09:35:30.553 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/solrw1:8983_solr
solr-overseer_1  | 2019-09-24 09:35:30.574 INFO  (zkCallback-5-thread-8-processing-n:solr-overseer:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
solrw1_1         | 2019-09-24 09:35:30.589 INFO  (zkCallback-5-thread-6-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
solrw1_1         | 2019-09-24 09:35:30.726 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
solrw1_1         | 2019-09-24 09:35:30.727 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.ConnectionManager Connected:true
solrw1_1         | 2019-09-24 09:35:32.569 ERROR (qtp762152757-8797) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:35:32.945 INFO  (qtp762152757-6501) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
solrw1_1         | 2019-09-24 09:35:32.986 INFO  (qtp762152757-6501) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=40
solrw1_1         | 2019-09-24 09:35:33.929 INFO  (qtp762152757-3634) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
solrw1_1         | 2019-09-24 09:35:33.947 INFO  (qtp762152757-3634) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=18
solr1_1          | Exception in thread "solr-idle-connections-evictor"
solr1_1          | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp762152757-9435"
solrw1_1         | 2019-09-24 09:36:09.080 ERROR (qtp762152757-8806) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr-overseer:8983/solr/kg1/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solrw1_1         | 2019-09-24 09:37:04.028 ERROR (qtp762152757-8805) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg2/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1         |
solr-overseer_1  | 2019-09-24 09:37:10.610 ERROR (qtp762152757-681) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solrw1:8983/solr/kg2/select
solr-overseer_1  | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solr-overseer_1  | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solr-overseer_1  | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solr-overseer_1  | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solr-overseer_1  | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solr-overseer_1  | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solr-overseer_1  | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solr-overseer_1  | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solr-overseer_1  | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solr-overseer_1  | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solr-overseer_1  | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solr-overseer_1  | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solr-overseer_1  | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solr-overseer_1  | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solr-overseer_1  | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solr-overseer_1  | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solr-overseer_1  | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solr-overseer_1  | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solr-overseer_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solr-overseer_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solr-overseer_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solr-overseer_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solr-overseer_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solr-overseer_1  | 	at java.lang.Thread.run(Thread.java:748)
solr-overseer_1  | Caused by: java.net.SocketTimeoutException: Read timed out
solr-overseer_1  | 	at java.net.SocketInputStream.socketRead0(Native Method)
solr-overseer_1  | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solr-overseer_1  | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solr-overseer_1  | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solr-overseer_1  | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solr-overseer_1  | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solr-overseer_1  | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solr-overseer_1  | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solr-overseer_1  | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solr-overseer_1  | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solr-overseer_1  | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solr-overseer_1  | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solr-overseer_1  | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solr-overseer_1  | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solr-overseer_1  | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solr-overseer_1  | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solr-overseer_1  | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solr-overseer_1  | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solr-overseer_1  | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solr-overseer_1  | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solr-overseer_1  | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solr-overseer_1  | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solr-overseer_1  | 	... 30 more
solr-overseer_1  |
solrw1_1         | 2019-09-24 09:37:10.631 ERROR (qtp762152757-8818) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg2/select
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
solrw1_1         | 	... 30 more
solrw1_1   

Logs after adding warning messages here :

solr1_1          | 2019-09-25 19:39:19.597 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.597 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.599 WARN  (qtp762152757-4915) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.599 WARN  (qtp762152757-4915) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.604 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.604 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.608 WARN  (qtp762152757-4916) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.608 WARN  (qtp762152757-4916) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.611 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.611 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.615 WARN  (qtp762152757-4917) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.615 WARN  (qtp762152757-4917) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.618 WARN  (qtp762152757-4923) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.618 WARN  (qtp762152757-4923) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.622 WARN  (qtp762152757-4919) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.622 WARN  (qtp762152757-4919) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.624 WARN  (qtp762152757-4924) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.625 WARN  (qtp762152757-4924) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.627 WARN  (qtp762152757-4918) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.627 WARN  (qtp762152757-4918) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.629 WARN  (qtp762152757-4925) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.629 WARN  (qtp762152757-4925) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.631 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.631 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.636 WARN  (qtp762152757-4926) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.636 WARN  (qtp762152757-4926) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.638 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.639 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
solr1_1          | 2019-09-25 19:39:19.641 WARN  (qtp762152757-4927) [   ] o.a.s.s.HttpSolrCall active core: null
solr1_1          | 2019-09-25 19:39:19.641 WARN  (qtp762152757-4927) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
solrw1_1         | 2019-09-25 19:39:19.645 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall active core: null
solrw1_1         | 2019-09-25 19:39:19.645 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2

Sample logs from master build:

solr1_1          | 2019-09-30 15:33:27.567 DEBUG (qtp1291367132-2817) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@10ba04e1{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.580 DEBUG (qtp1555990397-2810) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@16af72bc{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=1}
solr1_1          | 2019-09-30 15:33:27.595 DEBUG (qtp1291367132-2818) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@5ff7076f{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.603 DEBUG (qtp1555990397-2811) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@7fb4bb77{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.632 DEBUG (qtp1555990397-2813) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@703871b2{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.615 DEBUG (qtp1291367132-2819) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@55a5054b{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.669 DEBUG (qtp1555990397-2812) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/admin/metrics on HttpChannelOverHttp@31a022d8{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/admin/metrics,age=0}
solr1_1          | 2019-09-30 15:33:27.674 DEBUG (qtp1291367132-2820) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@51b1a1a5{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.686 DEBUG (qtp1555990397-2814) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/admin/metrics on HttpChannelOverHttp@31a022d8{r=2,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/admin/metrics,age=0}
solrw1_1         | 2019-09-30 15:33:27.699 DEBUG (qtp1555990397-2814) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@3337d99e{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.705 DEBUG (qtp1291367132-2821) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@3ecacfcf{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr-overseer_1  | 2019-09-30 15:33:27.710 DEBUG (qtp1291367132-23) [   ] o.e.j.s.HttpChannel REQUEST for //solr-overseer:8983/solr/admin/metrics on HttpChannelOverHttp@2c537da3{r=3,c=false,c=false/false,a=IDLE,uri=//solr-overseer:8983/solr/admin/metrics,age=0}
solrw1_1         | 2019-09-30 15:33:27.774 DEBUG (qtp1555990397-2812) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@31a022d8{r=3,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.814 DEBUG (qtp1555990397-2815) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@367478e4{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.796 DEBUG (qtp1291367132-2823) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/admin/metrics on HttpChannelOverHttp@4c72c6ae{r=5,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/admin/metrics,age=0}
solrw1_1         | 2019-09-30 15:33:27.848 DEBUG (qtp1555990397-2816) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@413c454d{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.797 DEBUG (qtp1291367132-2822) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@5d74692a{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.806 DEBUG (qtp1291367132-2824) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/admin/metrics on HttpChannelOverHttp@4c72c6ae{r=6,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/admin/metrics,age=0}
solrw1_1         | 2019-09-30 15:33:27.890 DEBUG (qtp1555990397-2817) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@23efdf4f{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.825 DEBUG (qtp1291367132-2823) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@5b439ce5{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solr1_1          | 2019-09-30 15:33:27.867 DEBUG (qtp1291367132-2824) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@7e1f0c8f{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
solrw1_1         | 2019-09-30 15:33:27.918 DEBUG (qtp15559

Issue Links

• duplicates: SOLR-13765 Deadlock on Solr cloud request causing 'Too many open files' error
• links to: GitHub Pull Request #922