[LUCENE-2357] Reduce transient RAM usage while merging by using packed ints array for docID re-mapping

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0-ALPHA, 6.0
• Component/s: core/index
• Lucene Fields: New
• Assignee: Adrien Grand
• Reporter: Michael McCandless
• Created: 29/Mar/10 11:20
• Updated: 28/Aug/22 12:23
• Resolved: 12/Jun/12 10:50

Description

We allocate this int[] to remap docIDs due to compaction of deleted ones.

This uses alot of RAM for large segment merges, and can fail to allocate due to fragmentation on 32 bit JREs.

Now that we have packed ints, a simple fix would be to use a packed int array... and maybe instead of storing abs docID in the mapping, we could store the number of del docs seen so far (so the remap would do a lookup then a subtract). This may add some CPU cost to merging but should bring down transient RAM usage quite a bit.

Issue Links

• relates to: LUCENE-4792 Smaller doc maps