randomizedtesting 2.0.8 has some tweaks to make it work. I tend to believe a "real" solution doesn't exist, this is the best I could come up with – there is a preallocated last-resort memory pool of 1MB that gets released when OOM is thrown. A small delay is introduced to allow concurrent GC threads to do their job, then the process attempts to serialize the exception and write back some messages about the OOM.
It still doesn't always work: I had some random OOMs even in System.out.printlns on J9, whether the GC does or doesn't reclaim the released memory is unknown and is an inherent race condition, the VM itself (hotspot) seems to require some memory when it does Runtime.halt; under low-memory conditions it always returns with exit code 1, regardless of the status passed to halt.
PermGen space errors are even more difficult. I tried preloading classes at startup but this turns out to be such an unholy mess (and still without the guarantee that something is not omitted) that I reverted this change. To my defense – I tried with Maven's surefire and it also craps out (hangs) under such conditions.