[LUCENE-2467] IndexWriter memory leak when large docs are indexed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3, 2.3.1, 2.3.2, 2.4, 2.4.1, 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2, 3.1, 4.0-ALPHA
• Fix Version/s: 2.9.3, 3.0.2, 3.1, 4.0-ALPHA
• Component/s: core/index
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Michael McCandless
• Created: 17/May/10 16:06
• Updated: 02/Sep/24 08:15
• Resolved: 26/May/10 19:08

Description

Spinoff from the java-user thread "IndexWriter and memory usage"...

IndexWriter has had a long standing memory leak, since LUCENE-843.

When the byte/char/int blocks are recycled to the common pool, the
per-thread DW classes incorrectly still hold a reference to them.

This normally is not a problem, since these buffers will be re-used
again.

But, if you index a massive document, causing IW to allocate more than
the RAM buffer allocated to it, then the leak happens. So you could
have a 16 MB RAM buffer set, but if a huge doc required allocation of
200 MB worth of arrays, those 200 MB are never freed (well, until you
close the IW and deref it from the app).

It's even worse if you use multiple threads: if each thread has ever
had to index a massive document, then that thread incorrectly holds
onto the extra arrays.