[SOLR-13862] JDK 13+Shenandoah stability/recovery problems

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 8.2
• Assignee: Unassigned
• Reporter: Bernd Wahlen
• Created: 23/Oct/19 10:39
• Updated: 23/Sep/20 08:33
• Resolved: 11/May/20 08:52

Description

after updating my cluster (centos 7.7, solr 8.2, jdk12) to JDK13 (3 nodes, 4 collections, 1 shard) everything was running good (with lower p95) for some hours. Then 2 nodes (not the leader) going to recovery state, but ~"Recovery failed Error opening new searcher". I tried rolling restart the cluster, but recovery is not working. After i switched to jdk11 recovery works again. In summary jdk11 or jdk12 was running stable, jdk13 not.

This is my solr.in.sh:

GC_TUNE="-XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC"
SOLR_TIMEZONE="CET"
GC_LOG_OPTS="-Xlog:gc*:file=/var/log/solr/solr_gc.log:time:filecount=9,filesize=20M:safepoint"

I also tried ADDREPLICA during my attempt to reapair the cluster, which causes Out of Memory on JDK 13 and worked after going back to JDK 11.