[SOLR-14314] Solr does not response most of the update request some times

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Invalid
• Assignee: Unassigned
• Reporter: Aaron Sun
• Created: 09/Mar/20 14:41
• Updated: 27/May/20 18:05
• Resolved: 13/Mar/20 11:13

Description

Solr version:

solr-spec
8.4.1
solr-impl
8.4.1 832bf13dd9187095831caf69783179d41059d013 - ishan - 2020-01-10 13:40:28
lucene-spec
8.4.1
lucene-impl
8.4.1 832bf13dd9187095831caf69783179d41059d013 - ishan - 2020-01-10 13:35:00

 

Java process:

java -Xms100G -Xmx200G -DSTOP.PORT=8078 -DSTOP.KEY=ardsolrstop -Dsolr.solr.home=/ardome/solr -Djetty.port=8983 -Dsolr.log.dir=/var/ardendo/log -jar start.jar --module=http

Run on a powerful server with 32 cores, 265GB RAM.

The problem is that time to time it start to get very slow to update solr documents, for example time out after 30 minutes.

document size is around 20k~50K each, each http request send to /update is around 4MB~10MB.

/update request is done by multi processes.

Some of the update get response, but the differences between "QTime"  and http response time is big, one example, qtime = 66s, http response time is 2304s.

According to jstack for the thread state, lots of BLOCKED state.

thread dump log is attached.

Any hint would be appreciate, thanks!