[SOLR-10140] Performance degradation and CPU spike when moving to Solr 6.4.1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 6.4.1
• Component/s: Server

Environment

CentOS Linux release 7.3.1611 (Core)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

• Assignee: Unassigned
• Reporter: bidorbuy
• Created: 15/Feb/17 10:46
• Updated: 08/Jun/19 15:31
• Resolved: 15/Feb/17 10:58

Description

We migrated from a perfectly healthy Solr 6.2.0 installation to Solr 6.4.1 and when switching Solr 6.4.1 into production, load-average and CPU trash - compare Solr6.4.1.png (which shows that CPU and load-average spike) with Solr6.2.0.png (very stable, predictable utilisation).

Migration changes:

• Changed luceneMatchVersion from 6.2.0 to 6.4.1 and rebuild index
• Changed solr.SynonymFilterFactory to solr.SynonymFilterFactory
• Removed defaultSearchField and replaced with df in solrconfig.xml
• Removed solrQueryParser defaultOperator and replaced with q.op in solrconfig.xml
• Increased heap from 3G to 4G via "SOLR_JAVA_MEM="-Xms4G -Xmx4G""
• Our GC tune remains unchanged:

  GC_TUNE="-XX:NewRatio=3 \
  -XX:SurvivorRatio=4 \
  -XX:TargetSurvivorRatio=90 \
  -XX:MaxTenuringThreshold=8 \
  -XX:+UseConcMarkSweepGC \
  -XX:+UseParNewGC \
  -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
  -XX:+CMSScavengeBeforeRemark \
  -XX:PretenureSizeThreshold=64m \
  -XX:+UseCMSInitiatingOccupancyOnly \
  -XX:CMSInitiatingOccupancyFraction=50 \
  -XX:CMSMaxAbortablePrecleanTime=6000 \
  -XX:+CMSParallelRemarkEnabled \
  -XX:+ParallelRefProcEnabled"
  

• I noticed in jetty.xml new additions of "InstrumentedQueuedThreadPool" and "InstrumentedHandler" - unsure if this would affect CPU overloading?

Since our production load is fairly static with regards to the index size (see Solr6.4.1-info-*.pngs) and since our current Solr 6.2.0 runs perfectly fine on the same version of CentOS and JDK I can only think that a change in Jetty or Solr/Lucene causes the trashing of CPU.

I would like to assist with isolating/resolving the issue but am not sure what other diagnostic information is needed (nor have I seen similar reports elsewhere).

Issue Links

• duplicates: SOLR-10130 Serious performance degradation in Solr 6.4.1 due to the new metrics collection