I think this is the reason people see long blocking periods under write load.
Currently when we hit the global memstore limit, we call reclaimMemStoreMemory() which is synchronized - thus everyone has to wait until the memory has flushed down to the low water mark. This causes every writer to block for 10-15 seconds on a large heap.
Instead we should start triggering flushes (in another thread) whenever we're above the low water mark. Then only block writers when we're above the high water mark.