[CASSANDRA-18125] Improve memtable allocator accounting when updating AtomicBTreePartition - ASF JIRA

Agile Board

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Normal
Resolution: Fixed
Fix Version/s: 4.0.9, 4.1.1, 5.0-alpha1, 5.0
Component/s: Local/Memtable
Labels:
None

Bug Category:
Correctness - Recoverable Corruption / Loss
Severity:
Normal
Complexity:
Challenging
Discovered By:
User Report
Platform:

All
Impacts:

None
Since Version:

4.0.5
Source Control Link:

https://github.com/apache/cassandra/commit/40f9ca60f103783aa481bc9a91b92fd55b4ea625
Test and Documentation Plan:

Hide

Test included that reproduces issue and checks fixed.

Show
Test included that reproduces issue and checks fixed.

Description

On two nodes (on a 5 nodes cluster) on the cluster I'm running, I got the following exception. It occurred at 3,5 minutes interval.

MemtableReclaimMemory:2625 org.apache.cassandra.service.CassandraDaemon uncaughtException - Exception in thread Thread[MemtableReclaimMemory:2625,5,main]java.lang.AssertionError: null
	at org.apache.cassandra.utils.memory.MemtablePool$SubPool.released(MemtablePool.java:193)
	at org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.releaseAll(MemtableAllocator.java:151)
	at org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.setDiscarded(MemtableAllocator.java:142)
	at org.apache.cassandra.utils.memory.MemtableAllocator.setDiscarded(MemtableAllocator.java:93)
	at org.apache.cassandra.utils.memory.SlabAllocator.setDiscarded(SlabAllocator.java:120)
	at org.apache.cassandra.db.Memtable.setDiscarded(Memtable.java:201)
	at org.apache.cassandra.db.ColumnFamilyStore$Flush$1.runMayThrow(ColumnFamilyStore.java:1216)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

$ nodetool info
ID                     : 
Gossip active          : true
Native Transport active: true
Load                   : 204.67 GiB
Generation No          : 1670343179
Uptime (seconds)       : 1110514
Heap Memory (MB)       : 7218.07 / 24576.00
Off Heap Memory (MB)   : 784.06
Data Center            : par
Rack                   : e1
Exceptions             : 1
Key Cache              : entries 802712, size 100 MiB, capacity 100 MiB, 774541004 hits, 914207516 requests, 0.847 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Percent Repaired       : 2.3272298419424144E-5%
Token                  : (invoke with -T/--tokens to see all 8 tokens)

$ java -version
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el7_9) (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.16.0.8-1.el7_9) (build 11.0.16+8-LTS, mixed mode, sharing)
$ nodetool version
ReleaseVersion: 4.0.6