Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-13793

HTTPSolrCall makes cascading calls even when all replicas are down for a collection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 6.6, master (9.0)
    • Fix Version/s: 7.7.3, 8.3
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      REMOTEQUERY action in HTTPSolrCall ends up making too many cascading remoteQuery calls when all all the replicas of a collection are in down state.
      This results in increase in thread count, unresponsive solr nodes and eventually node (one's which have this collection) going out of live nodes.

      Example scenario: Consider a cluster with 3 nodes(solr1, solrw1, solr-overseer1). A collection is present on solr1, solrw1 but both replicas are in down state. When a search request is made to solr-overseer1, since replica is not present locally a remote query is made to solr1 (we also consider inactive slices/coreUrls), solr1 also doesn't see an active replica present locally, it forwards to solrw1, again solrw1 will forward request to solr1. This goes on till both of solr1, solrw1 become unresponsive. Attached logs for this.

      This is happening because we are considering inactive slices, inactive coreUrl while forwarding requests to nodes.

      Steps to reproduce:

      1. Bring down all replicas of a collection but ensure nodes containing them are up
      2. Make any search call to any of solr nodes for this collection.

      Possible fixes:

      1. Ensure we select only active slices/coreUrls before making remote queries
      2. Put a limit on cascading calls probably limit to number of replicas
       
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:14.458 ERROR (qtp762152757-8772) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:14.451 ERROR (qtp762152757-8774) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:14.333 ERROR (qtp762152757-8771) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:12.661 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
      solrw1_1         | 2019-09-24 09:35:12.284 ERROR (qtp762152757-8767) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:12.275 ERROR (qtp762152757-8768) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:12.274 ERROR (qtp762152757-8766) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:12.054 ERROR (qtp762152757-8764) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:12.049 ERROR (qtp762152757-8763) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:12.044 ERROR (qtp762152757-8762) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:11.826 ERROR (qtp762152757-556) [   ] o.a.s.s.HttpSolrCall null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
      solrw1_1         | 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
      solrw1_1         | 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      solrw1_1         | 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
      solrw1_1         | 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:323)
      solrw1_1         | 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:320)
      solrw1_1         | 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
      solrw1_1         | 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:320)
      solrw1_1         | 	at org.apache.solr.handler.admin.ClusterStatus.getClusterStatus(ClusterStatus.java:82)
      solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$20(CollectionsHandler.java:664)
      solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:922)
      solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:228)
      solrw1_1         | 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215)
      solrw1_1         | 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:749)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:730)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         |
      zook1_1          | 2019-09-24 09:35:30,491 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16d6294ad750020 type:delete cxid:0xd zxid:0x9a5 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/election/102846757204197386-solrw1:8983_solr-n_0000000036 Error:KeeperErrorCode = NoNode for /solr/overseer_elect/election/102846757204197386-solrw1:8983_solr-n_0000000036
      zook1_1          | 2019-09-24 09:35:30,527 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@648] - Got user-level KeeperException when processing sessionid:0x16d6294ad750020 type:create cxid:0x12 zxid:0x9a7 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
      solrw1_1         | 2019-09-24 09:35:30.553 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/solrw1:8983_solr
      solr-overseer_1  | 2019-09-24 09:35:30.574 INFO  (zkCallback-5-thread-8-processing-n:solr-overseer:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
      solrw1_1         | 2019-09-24 09:35:30.589 INFO  (zkCallback-5-thread-6-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
      solrw1_1         | 2019-09-24 09:35:30.726 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
      solrw1_1         | 2019-09-24 09:35:30.727 INFO  (zkCallback-5-thread-4-processing-n:solrw1:8983_solr) [   ] o.a.s.c.c.ConnectionManager Connected:true
      solrw1_1         | 2019-09-24 09:35:32.569 ERROR (qtp762152757-8797) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg3/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:35:32.945 INFO  (qtp762152757-6501) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
      solrw1_1         | 2019-09-24 09:35:32.986 INFO  (qtp762152757-6501) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=40
      solrw1_1         | 2019-09-24 09:35:33.929 INFO  (qtp762152757-3634) [   ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
      solrw1_1         | 2019-09-24 09:35:33.947 INFO  (qtp762152757-3634) [   ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=18
      solr1_1          | Exception in thread "solr-idle-connections-evictor"
      solr1_1          | Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp762152757-9435"
      solrw1_1         | 2019-09-24 09:36:09.080 ERROR (qtp762152757-8806) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr-overseer:8983/solr/kg1/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solrw1_1         | 2019-09-24 09:37:04.028 ERROR (qtp762152757-8805) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg2/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1         |
      solr-overseer_1  | 2019-09-24 09:37:10.610 ERROR (qtp762152757-681) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solrw1:8983/solr/kg2/select
      solr-overseer_1  | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solr-overseer_1  | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solr-overseer_1  | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solr-overseer_1  | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solr-overseer_1  | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solr-overseer_1  | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solr-overseer_1  | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solr-overseer_1  | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solr-overseer_1  | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solr-overseer_1  | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solr-overseer_1  | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solr-overseer_1  | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solr-overseer_1  | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solr-overseer_1  | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solr-overseer_1  | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solr-overseer_1  | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solr-overseer_1  | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solr-overseer_1  | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solr-overseer_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solr-overseer_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solr-overseer_1  | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solr-overseer_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solr-overseer_1  | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solr-overseer_1  | 	at java.lang.Thread.run(Thread.java:748)
      solr-overseer_1  | Caused by: java.net.SocketTimeoutException: Read timed out
      solr-overseer_1  | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solr-overseer_1  | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solr-overseer_1  | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solr-overseer_1  | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solr-overseer_1  | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solr-overseer_1  | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solr-overseer_1  | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solr-overseer_1  | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solr-overseer_1  | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solr-overseer_1  | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solr-overseer_1  | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solr-overseer_1  | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solr-overseer_1  | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solr-overseer_1  | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solr-overseer_1  | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solr-overseer_1  | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solr-overseer_1  | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solr-overseer_1  | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solr-overseer_1  | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solr-overseer_1  | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solr-overseer_1  | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solr-overseer_1  | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solr-overseer_1  | 	... 30 more
      solr-overseer_1  |
      solrw1_1         | 2019-09-24 09:37:10.631 ERROR (qtp762152757-8818) [   ] o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error trying to proxy request for url: http://solr1:8983/solr/kg2/select
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:660)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:514)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
      solrw1_1         | 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      solrw1_1         | 	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
      solrw1_1         | 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
      solrw1_1         | 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
      solrw1_1         | 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
      solrw1_1         | 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
      solrw1_1         | 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
      solrw1_1         | 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
      solrw1_1         | 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      solrw1_1         | 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
      solrw1_1         | 	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
      solrw1_1         | 	at java.lang.Thread.run(Thread.java:748)
      solrw1_1         | Caused by: java.net.SocketTimeoutException: Read timed out
      solrw1_1         | 	at java.net.SocketInputStream.socketRead0(Native Method)
      solrw1_1         | 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      solrw1_1         | 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      solrw1_1         | 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      solrw1_1         | 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
      solrw1_1         | 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      solrw1_1         | 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      solrw1_1         | 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
      solrw1_1         | 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
      solrw1_1         | 	at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      solrw1_1         | 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      solrw1_1         | 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
      solrw1_1         | 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
      solrw1_1         | 	at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:632)
      solrw1_1         | 	... 30 more
      solrw1_1   
      

      Logs after adding warning messages here :

      solr1_1          | 2019-09-25 19:39:19.597 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.597 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.599 WARN  (qtp762152757-4915) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.599 WARN  (qtp762152757-4915) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.604 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.604 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.608 WARN  (qtp762152757-4916) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.608 WARN  (qtp762152757-4916) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.611 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.611 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.615 WARN  (qtp762152757-4917) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.615 WARN  (qtp762152757-4917) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.618 WARN  (qtp762152757-4923) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.618 WARN  (qtp762152757-4923) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.622 WARN  (qtp762152757-4919) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.622 WARN  (qtp762152757-4919) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.624 WARN  (qtp762152757-4924) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.625 WARN  (qtp762152757-4924) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.627 WARN  (qtp762152757-4918) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.627 WARN  (qtp762152757-4918) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.629 WARN  (qtp762152757-4925) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.629 WARN  (qtp762152757-4925) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.631 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.631 WARN  (qtp762152757-4920) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.636 WARN  (qtp762152757-4926) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.636 WARN  (qtp762152757-4926) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.638 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.639 WARN  (qtp762152757-4921) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      solr1_1          | 2019-09-25 19:39:19.641 WARN  (qtp762152757-4927) [   ] o.a.s.s.HttpSolrCall active core: null
      solr1_1          | 2019-09-25 19:39:19.641 WARN  (qtp762152757-4927) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solrw1:8983/solr/kg2
      solrw1_1         | 2019-09-25 19:39:19.645 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall active core: null
      solrw1_1         | 2019-09-25 19:39:19.645 WARN  (qtp762152757-4922) [   ] o.a.s.s.HttpSolrCall Will attempt to connect inactive core: http://solr1:8983/solr/kg2
      

      Sample logs from master build:

      solr1_1          | 2019-09-30 15:33:27.567 DEBUG (qtp1291367132-2817) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@10ba04e1{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.580 DEBUG (qtp1555990397-2810) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@16af72bc{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=1}
      solr1_1          | 2019-09-30 15:33:27.595 DEBUG (qtp1291367132-2818) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@5ff7076f{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.603 DEBUG (qtp1555990397-2811) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@7fb4bb77{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.632 DEBUG (qtp1555990397-2813) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@703871b2{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.615 DEBUG (qtp1291367132-2819) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@55a5054b{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.669 DEBUG (qtp1555990397-2812) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/admin/metrics on HttpChannelOverHttp@31a022d8{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/admin/metrics,age=0}
      solr1_1          | 2019-09-30 15:33:27.674 DEBUG (qtp1291367132-2820) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@51b1a1a5{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.686 DEBUG (qtp1555990397-2814) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/admin/metrics on HttpChannelOverHttp@31a022d8{r=2,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/admin/metrics,age=0}
      solrw1_1         | 2019-09-30 15:33:27.699 DEBUG (qtp1555990397-2814) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@3337d99e{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.705 DEBUG (qtp1291367132-2821) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@3ecacfcf{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr-overseer_1  | 2019-09-30 15:33:27.710 DEBUG (qtp1291367132-23) [   ] o.e.j.s.HttpChannel REQUEST for //solr-overseer:8983/solr/admin/metrics on HttpChannelOverHttp@2c537da3{r=3,c=false,c=false/false,a=IDLE,uri=//solr-overseer:8983/solr/admin/metrics,age=0}
      solrw1_1         | 2019-09-30 15:33:27.774 DEBUG (qtp1555990397-2812) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@31a022d8{r=3,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.814 DEBUG (qtp1555990397-2815) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@367478e4{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.796 DEBUG (qtp1291367132-2823) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/admin/metrics on HttpChannelOverHttp@4c72c6ae{r=5,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/admin/metrics,age=0}
      solrw1_1         | 2019-09-30 15:33:27.848 DEBUG (qtp1555990397-2816) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@413c454d{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.797 DEBUG (qtp1291367132-2822) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@5d74692a{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.806 DEBUG (qtp1291367132-2824) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/admin/metrics on HttpChannelOverHttp@4c72c6ae{r=6,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/admin/metrics,age=0}
      solrw1_1         | 2019-09-30 15:33:27.890 DEBUG (qtp1555990397-2817) [   ] o.e.j.s.HttpChannel REQUEST for //solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@23efdf4f{r=1,c=false,c=false/false,a=IDLE,uri=//solrw1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.825 DEBUG (qtp1291367132-2823) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@5b439ce5{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solr1_1          | 2019-09-30 15:33:27.867 DEBUG (qtp1291367132-2824) [   ] o.e.j.s.HttpChannel REQUEST for //solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json on HttpChannelOverHttp@7e1f0c8f{r=1,c=false,c=false/false,a=IDLE,uri=//solr1:8983/solr/kg1/select?q=*%3A*&indent=on&timeAllowed=5000&wt=json,age=0}
      solrw1_1         | 2019-09-30 15:33:27.918 DEBUG (qtp15559
      

        Attachments

        1. SOLR-13793.patch
          11 kB
          Kesharee Nandan Vishwakarma
        2. SOLR-13793.patch
          5 kB
          Kesharee Nandan Vishwakarma

          Issue Links

            Activity

              People

              • Assignee:
                ichattopadhyaya Ishan Chattopadhyaya
                Reporter:
                keshareenv Kesharee Nandan Vishwakarma
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h