[SOLR-13293] ConcurrentUpdateHttp2SolrClient always log AsynchronousCloseException exception error on indexing

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 8.0
• Fix Version/s: 8.3
• Component/s: SolrJ
• Assignee: Cao Manh Dat
• Reporter: Karl Stoney
• Created: 04/Mar/19 18:53
• Updated: 10/Oct/19 12:15
• Resolved: 09/Oct/19 20:01

Description

Hi,
Testing out branch_8x, we're randomly seeing the following errors on a simple 3 node cluster. It doesn't appear to affect replication (the cluster remains green).

They come in (mass, literally 1000s at a time) bulk.

There we no network issues at the time.

16:53:01.492 [updateExecutor-4-thread-34-processing-x:at-uk_shard1_replica_n1 r:core_node3 null n:solr-2.search-solr.preprod.k8.atcloud.io:80_solr c:at-uk s:shard1] ERROR org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient - Error consuming and closing http response stream.
java.nio.channels.AsynchronousCloseException: null
        at org.eclipse.jetty.client.util.InputStreamResponseListener$Input.read(InputStreamResponseListener.java:316) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
        at java.io.InputStream.read(InputStream.java:101) ~[?:1.8.0_191]
        at org.eclipse.jetty.client.util.InputStreamResponseListener$Input.read(InputStreamResponseListener.java:287) ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:283) ~[solr-solrj-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT b14748e61fd147ea572f6545265b883fa69ed27f - root
- 2019-03-04 16:30:04]
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176) ~[solr-solrj-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT b14748e61fd147ea572f6545265b883fa69ed27f - root - 2019-03-04
16:30:04]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) ~[metrics-core-3.2.6.jar:3.2.6]
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[solr-solrj-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT b14748e61fd147ea572f6545265b883fa69ed27f - root - 2019-03-04 16:30:04]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]