In a previous application, we'd set aside a bit of memory to release when application hit an OOME. The reservation was done on startup. It was a linked list of sizeable blocks rather than one big monolithic block, probably so allocation would still work in a fragmented heap. In that apps' case, default was a single block of 5M. Maybe in hbase, set aside more? 20M? In 4 blocks? Make it configurable? What you think? Default hbase heap is 1G I believe (See the bin/hbase script)
Main loop in the application was wrapped in a try/catch. On 'serious error', we'd first release the memory resevoir – the release could be run by more than one thread so some times it'd be a noop – and then run code to set the application into a safe 'park' so could be analyzed later by operator. In our case, things would be a little trickier because there is more than just the one loop. The OOME could bubble out in the main master/regionserver loops or in one of the service thread loops. You'd have to plug in the OOME processing into all places (some inherit from Chore so you could add processing there). We also want our regionserver to go all the ways down if it hit an OOME to minimize damage done.
I'd imagine that all you'd do is if OOME, release the memory and then let the shutdown proceed normally. Hopefully, the very release of the resevoir should be sufficient to making a successful shutdown.
Tests will be hard. There is an OOMERegionServer. You might play with that. You probably won't be able to inline it as unit test. Thats OK, I think. Also, I don't think its possible to write a handler that will work in all cases, just most: e.g. there may be a pathological case where the just-release resevoir gets eaten up immediately but a rampant thread. I think we'll just have to make up a patch that does the above, commit it and then watch how well it does out in the field.