Patch. I used RamUsageEstimator to estimate the memory savings depending on the ratio of deleted documents on a segment of 100M docs (deleted docs are randomly chosen):

+1, this is great.

[trunk commit] Adrien Grand
http://svn.apache.org/viewvc?view=revision&revision=1448853

LUCENE-4792: Reduction of the memory required to build the doc ID maps used when merging segments.

Committed. Thanks for the review Robert!

[branch_4x commit] Adrien Grand
http://svn.apache.org/viewvc?view=revision&revision=1448861

LUCENE-4792: Reduction of the memory required to build the doc ID maps used when merging segments (merged from r1448853).

These RAM savings are AWESOME! Where else can we use MonotonicAppendingLongBuffer!

We are using the same compression for (as far as i know):

• stored fields, term vectors, docvalues "disk" addresses
• multidocvalues ordinal maps

We could consider trying it out for fieldcache and other places for example, im not sure what the perf hit would be.
(I'm not very interested in optimizing fieldcache myself)

In case someone would like to use this class, I'd add that:

• the encoded sequence does not strictly need to be monotonic: it can encode any sequence of values but it compresses best when the stream contains monotonic sub-sequences of 1024 longs at least (for example it would have a good compression ratio if there are first 10000 increasing values and then 5000 decreasing values),
• it can address up to 2^42 values,
• there are writer/reader equivalents called MonotonicBlockPackedWriter and MonotonicBlockPackedReader (which can either load values in memory or read from disk).

Closed after release.