[CASSANDRA-19535] Running with default configuration OOMs under light load

• Type: Bug
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 5.0-beta1
• Assignee: Unassigned
• Reporter: Jon Haddad
• Created: 05/Apr/24 21:07
• Updated: 05/Apr/24 21:08

Description

Under very moderate load, Cassandra crashes with direct memory errors.  

ERROR [epollEventLoopGroup-5-16] 2024-04-04 19:16:50,096 JVMStabilityInspector.java:186 - Force heap space OutOfMemoryError in the presence of
java.lang.OutOfMemoryError: Cannot reserve 65536 bytes of direct buffer memory (allocated: 8589921806, limit: 8589934592)
	at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121)
	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332)
	at org.apache.cassandra.utils.memory.BufferPool.allocate(BufferPool.java:238)
	at org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:923)
	at org.apache.cassandra.utils.memory.BufferPool$LocalPool.getAtLeast(BufferPool.java:901)
	at org.apache.cassandra.utils.memory.BufferPool.getAtLeast(BufferPool.java:219)
	at org.apache.cassandra.net.BufferPoolAllocator.getAtLeast(BufferPoolAllocator.java:75)
	at org.apache.cassandra.net.BufferPoolAllocator.newDirectBuffer(BufferPoolAllocator.java:63)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
	at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)