[CASSANDRA-15367] Memtable memory allocations may deadlock

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.21, 3.11.7, 4.0-alpha4, 4.0
• Component/s: Local/Commit Log, Local/Memtable
• Bug Category: Availability - Process Crash
• Severity: Critical
• Complexity: Challenging
• Discovered By: User Report
• Platform: All
• Since Version: 2.1 beta1

Source Control Link

e3f54d4a0c3403141db24f86714c3900eb9f212e

Test and Documentation Plan

unit test included

• Assignee: Benedict Elliott Smith
• Reporter: Benedict Elliott Smith

Authors

Benedict Elliott Smith

Reviewers

Benedict Elliott Smith, Blake Eggleston

• Created: 22/Oct/19 11:32
• Updated: 15/May/20 08:41
• Resolved: 08/Apr/20 00:03

Description

• Under heavy contention, we guard modifications to a partition with a mutex, for the lifetime of the memtable.
• Memtables block for the completion of all OpOrder.Group started before their flush began
• Memtables permit operations from this cohort to fall-through to the following Memtable, in order to guarantee a precise commitLogUpperBound
• Memtable memory limits may be lifted for operations in the first cohort, since they block flush (and hence block future memory allocation)

With very unfortunate scheduling

• A contended partition may rapidly escalate to a mutex
• The system may reach memory limits that prevent allocations for the new Memtable’s cohort (C2)
• An operation from C2 may hold the mutex when this occurs
• Operations from a prior Memtable’s cohort (C1), for a contended partition, may fall-through to the next Memtable
• The operations from C1 may execute after the above is encountered by those from C2