Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15358

LARGE_MESSAGE connection allocates heap buffer when BufferPool exhausted

    XMLWordPrintableJSON

    Details

      Description

      Hitting a bug with cassandra 4 alpha version. The same bug is repeated with difefrent version of Java(8,11 &12) Benedict Elliott Smith

       

      Stack trace:

      INFO [main] 2019-10-11 16:07:12,024 Server.java:164 - Starting listening for CQL clients on /1.3.0.6:9042 (unencrypted)...
      WARN [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:343 - CassandraRoleManager skipped default role setup: some nodes were not ready
      INFO [OptionalTasks:1] 2019-10-11 16:07:13,961 CassandraRoleManager.java:369 - Setup task failed with error, rescheduling
      WARN [Messaging-EventLoop-3-2] 2019-10-11 16:07:22,038 NoSpamLogger.java:94 - 10.3x.4x.5x:7000->1.3.0.5:7000-LARGE_MESSAGES-[no-channel] dropping message of type PING_REQ whose timeout expired before reaching the network
      WARN [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:343 - CassandraRoleManager skipped default role setup: some nodes were not ready
      INFO [OptionalTasks:1] 2019-10-11 16:07:23,963 CassandraRoleManager.java:369 - Setup task failed with error, rescheduling
      INFO [Messaging-EventLoop-3-6] 2019-10-11 16:07:32,759 NoSpamLogger.java:91 - 10.3x.4x.5x:7000->1.3.0.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
      io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /1.3.0.2:7000
      Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
      at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
      at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
      at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:667)
      at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:644)
      at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524)
      at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:414)
      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      at java.base/java.lang.Thread.run(Thread.java:834)
      
      WARN [Messaging-EventLoop-3-3] 2019-10-11 16:11:32,639 NoSpamLogger.java:94 - 1.3.4.6:7000->1.3.4.5:7000-URGENT_MESSAGES-[no-channel] dropping message of type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
      INFO [Messaging-EventLoop-3-18] 2019-10-11 16:11:33,077 NoSpamLogger.java:91 - 1.3.4.5:7000->1.3.4.4:7000-URGENT_MESSAGES-[no-channel] failed to connect
       
      ERROR [Messaging-EventLoop-3-11] 2019-10-10 01:34:34,407 InboundMessageHandler.java:657 - 1.3.4.5:7000->1.3.4.8:7000-LARGE_MESSAGES-0b7d09cd unexpected exception caught while processing inbound messages; terminating connection
      java.lang.IllegalArgumentException: initialBuffer is not a direct buffer.
      at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:87)
      at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:59)
      at org.apache.cassandra.net.BufferPoolAllocator$Wrapped.<init>(BufferPoolAllocator.java:95)
      at org.apache.cassandra.net.BufferPoolAllocator.newDirectBuffer(BufferPoolAllocator.java:56)
      at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
      at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
      at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
      at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
      at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
      at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:777)
      at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:424)
      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:326)
      at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      at java.base/java.lang.Thread.run(Thread.java:835)
      

        Attachments

        1. verbose_logs.diff
          5 kB
          Valentin Lorentz
        2. verbose_logs.txt
          3 kB
          Valentin Lorentz
        3. all_errors.txt
          191 kB
          Valentin Lorentz
        4. repair_1_trace.txt
          5 kB
          Gaurav Ashok
        5. debug_logs_during_repair.txt
          23 kB
          Gaurav Ashok

          Activity

            People

            • Assignee:
              benedict Benedict Elliott Smith
              Reporter:
              SRAM85 Santhosh Kumar Ramalingam
              Authors:
              Benedict Elliott Smith
              Reviewers:
              David Capwell, Dinesh Joshi
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: