Here's an example: http://jenkins.sarowe.net/job/Lucene-core-nightly-monster-5.x-Java7/22/console; the seed still OOMs for me when -Dtests.heapsize=60g:
(Although the repro line doesn't include all of them - Dawid Weiss is this expected? - the above-linked job runs with -Dtests.jvms=4 -Dtests.nightly=true -Dtests.monster=true -Dtests.heapsize=30g.)
I narrowed the problem down to CompressingCodec - any of the 4 subclasses - with maxDocsPerChunk * blockSize less than 16. The other variable randomized in CompressingCodec.randomInstance() is chunkSize, which doesn't seem to be implicated in the OOMs.
I guess we could suppress CompressingCodec, or just expect OOMs from time to time (the above condition happens fairly rarely). But I'd rather continue to exercise CompressingCodec as much as possible.
Here are the OOMs I saw with -Dtests.heapsize=30g and -Dtests.codec=CompressingCodec:
Here are some that did not OOM: