[LUCENE-4512] Additional memory savings in CompressingStoredFieldsIndex.MEMORY_CHUNK

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.1, 6.0
• Lucene Fields: New
• Assignee: Adrien Grand
• Reporter: Adrien Grand
• Created: 29/Oct/12 14:34
• Updated: 28/Aug/22 13:30
• Resolved: 31/Oct/12 19:02

Description

Robert had a great idea to save memory with CompressingStoredFieldsIndex.MEMORY_CHUNK: instead of storing the absolute start pointers we could compute the mean number of bytes per chunk of documents and only store the delta between the actual value and the expected value (avgChunkBytes * chunkNumber).

By applying this idea to every n(=1024?) chunks, we would even:

• make sure to never hit the worst case (delta ~= maxStartPointer)
• reduce memory usage at indexing time.