[CASSANDRA-16318] Memtable heap size is severely underestimated

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-rc1, 4.0
• Component/s: Local/Memtable
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0-beta4

Source Control Link

https://github.com/apache/cassandra/commit/661f1aab171dc3ef16075f69581e88ad4a133fae

Test and Documentation Plan

https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/545/workflows/f797bf58-b572-4d8c-831e-a61936d23624

• Assignee: Ekaterina Dimitrova
• Reporter: Branimir Lambov

Authors

B Lambov (old handle), Ekaterina Dimitrova

Reviewers

Benjamin Lerer, B Lambov (old handle)

• Created: 09/Dec/20 09:04
• Updated: 25/Feb/21 10:25
• Resolved: 19/Jan/21 19:24

Description

We seem to be estimating the size of the on-heap memtable metadata to be around half of what it actually is. For example, during a read benchmark which writes 1 million single-long rows the memtable reports

1000000 ops, 58.174MiB serialized bytes, 385.284MiB (19%) on heap, 0.000KiB (0%) off-heap

while a heap dump taken at this point:

lists an usage of about 666MB altogether.

Switching to offheap_objects, the reported numbers are

1000000 ops, 58.174MiB serialized bytes, 233.650MiB (12%) on heap, 53.406MiB (3%) off-heap

while actual heap usage:

is about 442MB.

Looking at the code we definitely are not counting the AtomicBTreePartition.Holder, EncodingStats, liveness and deletion info objects associated with each partition, and most probably others.