[CASSANDRA-18125] Improve memtable allocator accounting when updating AtomicBTreePartition

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.9, 4.1.1, 5.0-alpha1, 5.0
• Component/s: Local/Memtable
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Normal
• Complexity: Challenging
• Discovered By: User Report
• Platform: All
• Since Version: 4.0.5

Source Control Link

https://github.com/apache/cassandra/commit/40f9ca60f103783aa481bc9a91b92fd55b4ea625

Test and Documentation Plan

Test included that reproduces issue and checks fixed.

• Assignee: Benedict Elliott Smith
• Reporter: Nicolas Henneaux

Authors

Benedict Elliott Smith

Reviewers

Benjamin Lerer, Jon Meredith

• Created: 19/Dec/22 12:46
• Updated: 12/Sep/23 19:18
• Resolved: 07/Mar/23 17:58

Description

On two nodes (on a 5 nodes cluster) on the cluster I'm running, I got the following exception. It occurred at 3,5 minutes interval.

MemtableReclaimMemory:2625 org.apache.cassandra.service.CassandraDaemon uncaughtException - Exception in thread Thread[MemtableReclaimMemory:2625,5,main]java.lang.AssertionError: null
	at org.apache.cassandra.utils.memory.MemtablePool$SubPool.released(MemtablePool.java:193)
	at org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.releaseAll(MemtableAllocator.java:151)
	at org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.setDiscarded(MemtableAllocator.java:142)
	at org.apache.cassandra.utils.memory.MemtableAllocator.setDiscarded(MemtableAllocator.java:93)
	at org.apache.cassandra.utils.memory.SlabAllocator.setDiscarded(SlabAllocator.java:120)
	at org.apache.cassandra.db.Memtable.setDiscarded(Memtable.java:201)
	at org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1216)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

$ nodetool info
ID                     : 
Gossip active          : true
Native Transport active: true
Load                   : 204.67 GiB
Generation No          : 1670343179
Uptime (seconds)       : 1110514
Heap Memory (MB)       : 7218.07 / 24576.00
Off Heap Memory (MB)   : 784.06
Data Center            : par
Rack                   : e1
Exceptions             : 1
Key Cache              : entries 802712, size 100 MiB, capacity 100 MiB, 774541004 hits, 914207516 requests, 0.847 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Percent Repaired       : 2.3272298419424144E-5%
Token                  : (invoke with -T/--tokens to see all 8 tokens)

$ java -version
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el7_9) (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.16.0.8-1.el7_9) (build 11.0.16+8-LTS, mixed mode, sharing)
$ nodetool version
ReleaseVersion: 4.0.6

Issue Links

• causes: CASSANDRA-18360 Test Failure: o.a.c.cql3.validation.operations.AlterTest#testDropListAndAddListWithSameName
• fixes: CASSANDRA-18253 Test Failures: dtest repair_tests.repair_test.TestRepair.test_simple_sequential_repair
• relates to: CASSANDRA-18159 Test Failure: repair_tests.repair_test.TestRepair.test_*dc_repair
• relates to: CASSANDRA-18266 dtest-offheap.repair_tests.repair_test.TestRepair.test_thread_count_repair
• relates to: CASSANDRA-18253 Test Failures: dtest repair_tests.repair_test.TestRepair.test_simple_sequential_repair