[LUCENE-1566] Large Lucene index can hit false OOM due to Sun JRE issue

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 2.9
• Component/s: core/index
• Lucene Fields: New
• Assignee: Simon Willnauer
• Reporter: Michael McCandless
• Created: 17/Mar/09 12:09
• Updated: 28/Aug/22 11:58
• Resolved: 16/Jul/09 18:08

Description

This is not a Lucene issue, but I want to open this so future google
diggers can more easily find it.

There's this nasty bug in Sun's JRE:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6478546

The gist seems to be, if you try to read a large (eg 200 MB) number of
bytes during a single RandomAccessFile.read call, you can incorrectly
hit OOM. Lucene does this, with norms, since we read in one byte per
doc per field with norms, as a contiguous array of length maxDoc().

The workaround was a custom patch to do large file reads as several
smaller reads.

Background here:

http://www.nabble.com/problems-with-large-Lucene-index-td22347854.html