[SOLR-16463] Serious crash on JDK17+ due to JIT on caffeinecache

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 9.0
• Fix Version/s: 9.1
• Component/s: Docker
• Assignee: Jan Høydahl
• Reporter: Jan Høydahl
• Created: 14/Oct/22 09:25
• Updated: 28/Sep/23 13:35
• Resolved: 20/Oct/22 08:03

Description

Solr 9+ under JDK 17+ may crash frequently, during JVM Hotspot optimization of caffeine cache class.

This is due to a JDK bug (https://bugs.openjdk.org/browse/JDK-8285835), but may not be fixed soon in JDK17, so we should make a workaround to protect our users. The bug is also reported in caffeine project (https://github.com/ben-manes/caffeine/issues/797).

So there are a few possible ways to avoid this

• Run Solr 9 on JDK 11
• Do not use caffeine cache, find some replacement
• Caffeine cache releases a new version that do not suffer the issue, and solr uses that
• Instruct JDK to not optimize that class, using JDK option 
  -XX:CompileCommand=exclude,com.github.benmanes.caffeine.cache.BoundedLocalCache::put

See users list for examples of this issue seen in the wild: https://lists.apache.org/thread/wg7qtkddd1t5h08okj7gm9qbrpdf0ox6 

Docker users can set SOLR_OPTS with the JDK option above. Patching the official Dockerfile to include this may be the least intrusive fix short-term. We should also document the issue on website and perhaps docker hub to provide users with a workaround.

For 9.1 we can hardcode the JDK flag in bin/solr.

Issue Links

• is superceded by: SOLR-17001 Adjust workaround for JDK17 bug so it only applies to specific versions with the bug
• links to: GitHub Pull Request #10
• links to: GitHub Pull Request #1090