[CASSANDRA-15358] LARGE_MESSAGE connection allocates heap buffer when BufferPool exhausted

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-alpha4, 4.0
• Component/s: Test/benchmark
• Labels: 4.0 alpha
• Bug Category: Availability - Response Crash
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/443bca18839268cd100930b380e0534b052a8c89

Test and Documentation Plan

n/a - entirely removed causative mechanism

• Assignee: Benedict Elliott Smith
• Reporter: Santhosh Kumar Ramalingam

Authors

Benedict Elliott Smith

Reviewers

David Capwell, Dinesh Joshi

• Created: 16/Oct/19 15:39
• Updated: 15/May/20 08:54
• Resolved: 10/Apr/20 00:32

Description

Hitting a bug with cassandra 4 alpha version. The same bug is repeated with difefrent version of Java(8,11 &12) benedict

 

Stack trace:

INFO [main] 2019-10-11 16:07:12,024 Server.java:164 - Starting listening for CQL clients on /1.3.0.6:9042 (unencrypted)...
WARN [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:343 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:369 - Setup task failed with error, rescheduling
WARN [Messaging-EventLoop-3-2] 2019-10-11 16:07:22,038 NoSpamLogger.java:94 - 10.3x.4x.5x:7000->1.3.0.5:7000-LARGE_MESSAGES-[no-channel] dropping message of type PING_REQ whose timeout expired before reaching the network
WARN [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:343 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:369 - Setup task failed with error, rescheduling
INFO [Messaging-EventLoop-3-6] 2019-10-11 16:07:32,759 NoSpamLogger.java:91 - 10.3x.4x.5x:7000->1.3.0.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /1.3.0.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:667)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:644)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)

WARN [Messaging-EventLoop-3-3] 2019-10-11 16:11:32,639 NoSpamLogger.java:94 - 1.3.4.6:7000->1.3.4.5:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
INFO [Messaging-EventLoop-3-18] 2019-10-11 16:11:33,077 NoSpamLogger.java:91 - 1.3.4.5:7000->1.3.4.4:7000-URGENT_MESSAGES-[no-channel] failed to connect
 
ERROR [Messaging-EventLoop-3-11] 2019-10-10 01:34:34,407 InboundMessageHandler.java:657 - 1.3.4.5:7000->1.3.4.8:7000-LARGE_MESSAGES-0b7d09cd unexpected exception caught while processing inbound messages; terminating connection
java.lang.IllegalArgumentException: initialBuffer is not a direct buffer.
at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:87)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:59)
at org.apache.cassandra.net.BufferPoolAllocator$Wrapped.<init>(BufferPoolAllocator.java:95)
at org.apache.cassandra.net.BufferPoolAllocator.newDirectBuffer(BufferPoolAllocator.java:56)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:777)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:835)