[LUCENE-4740] Weak references cause extreme GC churn

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.6.1
• Fix Version/s: 4.4, 6.0
• Component/s: core/store

Environment

Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16 cores
-Xmx16G

• Lucene Fields: New
• Assignee: Uwe Schindler
• Reporter: Kristofer Karlsson
• Created: 31/Jan/13 15:49
• Updated: 28/Aug/22 13:38
• Resolved: 14/Jun/13 17:55

Description

We are running a set of independent search machines, running our custom software using lucene as a search library. We recently upgraded from lucene 3.0.3 to 3.6.1 and noticed a severe degradation of performance.

After doing some heap dump digging, it turns out the process is stalling because it's spending so much time in GC. We noticed about 212 million WeakReference, originating from WeakIdentityMap, originating from MMapIndexInput.

Our problem completely went away after removing the clones weakhashmap from MMapIndexInput, and as a side-effect, disabling support for explictly unmapping the mmapped data.

Issue Links

• is depended upon by: JCR-3696 Upgrade Lucene to 4.4 or higher
• is duplicated by: LUCENE-4741 Weak references cause extreme GC churn
• relates to: LUCENE-4755 Unmap MMapIndexInput's in a delayed way, and avoid WeakReference usage.