[SOLR-5850] Race condition in ConcurrentUpdateSolrServer

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.6
• Component/s: clients - java, search, SolrCloud, update
• Labels: 500 cloud difficulty-medium error impact-medium update
• Assignee: Timothy Potter
• Reporter: Devansh Dhutia
• Created: 11/Mar/14 13:56
• Updated: 30/May/15 21:35

Description

Possibly related to SOLR-2308, we are seeing a Queue Full error message when issuing writes to Solr Cloud

Each Update has 200 documents, and a commit is issued after 2000 documents have been added.

The writes are spread out to all the servers in the cloud (2 in this case) and following is the stack trace from Solr:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">500</int><int name="QTime">101</int></lst><lst name="error"><str name="msg">Queue full</str><str name="t
race">java.lang.IllegalStateException: Queue full
        at java.util.AbstractQueue.add(Unknown Source)
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner$1.writeTo(ConcurrentUpdateSolrServer.java:181)
        at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:72)
        at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
        at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
        at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
        at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:232)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
</str><int name="code">500</int></lst>
</response>

Issue Links

• is related to: SOLR-7571 Return metrics with update requests to allow clients to self-throttle