-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Component/s: core/codecs
-
Labels:None
-
Lucene Fields:New
In SOLR-7927, I am trying to reduce the memory required to index very large documents (between 10 to 100MB) and one of the places which allocate a lot of heap is the UTF8 encoding in CompressingStoredFieldsWriter. The same problem existed in JavaBinCodec and we reduced its memory allocation by falling back to a double pass approach in SOLR-7971 when the utf8 size of the string is greater than 64KB.
I propose to make the same changes to CompressingStoredFieldsWriter as we made to JavaBinCodec in SOLR-7971.
- is required by
-
SOLR-7927 Indexing large documents requires larger heap than may be necessary
-
- Open
-