[LUCENE-2387] IndexWriter retains references to Readers used in Fields (memory leak)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 2.9.3, 3.0.2, 3.1, 4.0-ALPHA
• Lucene Fields: New
• Assignee: Michael McCandless
• Reporter: Ruben Laguna
• Created: 08/Apr/10 20:29
• Updated: 28/Aug/22 12:24
• Resolved: 04/May/10 23:45

Description

As described in [1] IndexWriter retains references to Reader used in Fields and that can lead to big memory leaks when using tika's ParsingReaders (as those can take 1MB per ParsingReader).

[2] shows a screenshot of the reference chain to the Reader from the IndexWriter taken with Eclipse MAT (Memory Analysis Tool) . The chain is the following:

IndexWriter -> DocumentsWriter -> DocumentsWriterThreadState -> DocFieldProcessorPerThread -> DocFieldProcessorPerField -> Fieldable -> Field (fieldsData)

-------------
[1] http://markmail.org/thread/ndmcgffg2mnwjo47
[2] http://skitch.com/ecerulm/n7643/eclipse-memory-analyzer

Issue Links

• relates to: LUCENE-8651 Tokenizer implementations can't be reset
• relates to: LUCENE-8650 ConcatenatingTokenStream does not end() nor reset() properly