[CASSANDRA-16201] Reduce amount of allocations during batch statement execution

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.10, 4.0-rc1, 4.0
• Component/s: Local/Other
• Bug Category: Degradation - Resource Management
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0 alpha 1

Source Control Link

https://github.com/apache/cassandra/commit/8333d0b0890f6299fac1fd219f58aee0c62cbf9c

Test and Documentation Plan

cci runs

• Assignee: Marcus Eriksson
• Reporter: Thomas Steinmaurer

Authors

Marcus Eriksson

Reviewers

Benjamin Lerer, Michael Semb Wever, Yifan Cai

• Created: 08/Oct/20 13:34
• Updated: 22/Jan/21 11:21
• Resolved: 22/Jan/21 10:03

Description

In a Cas 2.1 / 3.0 / 3.11 / 4.0b2 comparison test with the same load profile, we see 4.0b2 going OOM from time to time. According to a heap dump, we have multiple NTR threads in a 3-digit MB range.

This is likely related to object array pre-allocations at the size of BatchUpdatesCollector.updatedRows per BTree although there is always only 1 BTreeRow in the BTree.

So it seems we have many, many 20K elemnts pre-allocated object arrays resulting in a shallow heap of 80K each, although there is only one element in the array.

This sort of pre-allocation is causing a lot of memory pressure.

Issue Links

• Blocked: CASSANDRA-13900 Massive GC suspension increase after updating to 3.0.14 from 2.1.18
• is duplicated by: CASSANDRA-15430 Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18
• is related to: CASSANDRA-15430 Cassandra 3.0.18: BatchMessage.execute - 10x more on-heap allocations compared to 2.1.18