[CASSANDRA-17163] High CAS failures in MemtablePool.SubPool.tryAllocate

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Legacy/Local Write-Read Paths
• Bug Category: Degradation
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Benjamin Lerer
• Reporter: Benjamin Lerer

Authors

Benjamin Lerer

• Created: 18/Nov/21 13:45
• Updated: 10/Aug/23 08:11
• Resolved: 10/Aug/23 08:11

Description

In a cluster using memtable_allocation_type: offheap_buffers which handle a high amount of writes we are seeing some high contentions on MemtablePool.SubPool.tryAllocate.

"MutationStage-561" #5457 daemon prio=5 os_prio=0 tid=0x00007f35dd3cfa50 nid=0x1407a runnable [0x00007f2357fd5000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.cassandra.utils.memory.MemtablePool$SubPool.tryAllocate(MemtablePool.java:159)
	at org.apache.cassandra.utils.memory.MemtableAllocator$SubAllocator.allocate(MemtableAllocator.java:175)
	at org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:89)
	at org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:57)
	at org.apache.cassandra.utils.memory.ContextAllocator.clone(ContextAllocator.java:47)
	at org.apache.cassandra.db.rows.BufferCell.copy(BufferCell.java:140)
	at org.apache.cassandra.utils.memory.AbstractAllocator$CloningBTreeRowBuilder.addCell(AbstractAllocator.java:89)
	at org.apache.cassandra.db.rows.Rows.copy(Rows.java:49)
	at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:370)
	at org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:333)
	at org.apache.cassandra.utils.btree.BTree.buildLeaf(BTree.java:195)
	at org.apache.cassandra.utils.btree.BTree.buildInternal(BTree.java:208)
	at org.apache.cassandra.utils.btree.BTree.buildInternal(BTree.java:227)
	at org.apache.cassandra.utils.btree.BTree.buildInternal(BTree.java:254)
	at org.apache.cassandra.utils.btree.BTree.build(BTree.java:177)
	at org.apache.cassandra.utils.btree.BTree.update(BTree.java:283)
	at org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDeltaInternal(AtomicBTreePartition.java:143)
	at org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:184)
	at org.apache.cassandra.db.Memtable.put(Memtable.java:295)
	at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1335)
	at org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40)
	at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:661)
	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:513)
	at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
	at org.apache.cassandra.db.Mutation.apply(Mutation.java:220)
	at org.apache.cassandra.db.Mutation.apply(Mutation.java:229)
	at org.apache.cassandra.service.StorageProxy$$Lambda$1019/640627333.run(Unknown Source)
	at org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1544)
	at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable$1.runMayThrow(StorageProxy.java:2314)
	at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2352)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

This issue is similar to CASSANDRA-15922.

Issue Links

• duplicates: CASSANDRA-15511 Utilising BTree Improvements