A tserver experienced a OOME during minor compaction. This OOME was thrown because java could not create a native thread. Minor compactions only catch declared exceptions and RuntimeExceptions. This left the system in a state where the compaction was not running but the tserver thought it was. This cause"flush -w" to hang and prevented the tserver from reclaiming memory.
For whatever reason the OOME handler that kills the process did not kick in (seems it only kicks in w/ OOME related to heap allocation).