[LUCENE-6779] Reduce memory allocated by CompressingStoredFieldsWriter to write large strings

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 5.4, 6.0
• Component/s: core/codecs
• Lucene Fields: New
• Assignee: Shalin Shekhar Mangar
• Reporter: Shalin Shekhar Mangar
• Created: 03/Sep/15 20:20
• Updated: 09/May/16 18:28
• Resolved: 15/Sep/15 15:37

Description

In SOLR-7927, I am trying to reduce the memory required to index very large documents (between 10 to 100MB) and one of the places which allocate a lot of heap is the UTF8 encoding in CompressingStoredFieldsWriter. The same problem existed in JavaBinCodec and we reduced its memory allocation by falling back to a double pass approach in SOLR-7971 when the utf8 size of the string is greater than 64KB.

I propose to make the same changes to CompressingStoredFieldsWriter as we made to JavaBinCodec in SOLR-7971.

Issue Links

• is required by: SOLR-7927 Indexing large documents requires larger heap than may be necessary