[CASSANDRA-12358] Slow PostFlush execution due to 2i flushing can cause near OOM to OOM

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.10
• Component/s: Legacy/Core
• Severity: Normal
• Assignee: Ariel Weisberg
• Reporter: Ariel Weisberg

Authors

Ariel Weisberg

Reviewers

Branimir Lambov

• Created: 02/Aug/16 00:41
• Updated: 16/Apr/19 09:30
• Resolved: 09/Aug/16 16:01

Description

2i can be slow to flush for a variety of reasons. Potentially slower than the rate at which Memtables can ingest and flush data. If this occurs the heap fills up with Memtables that are waiting for PostFlush to run.

This occurs because reclaiming the memory is done before PostFlush runs.

I will post a branch that has the reclaim memory task run after PostFlush has completed. As far as I can tell this is safe and correct since the memory is committed up until that point.

It's not clear to me if PostFlush has to bind the Memtables or not. I suspect it does, but I'm not sure if that is a route I should go down.