Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15013

Message Flusher queue can grow unbounded, potentially running JVM out of memory

    Details

      Description

      This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue bounded, since, in the current state, items get added to the queue without any checks on queue size, nor with any checks on netty outbound buffer to check the isWritable state.
      We are seeing this issue hit our production 3.0 clusters quite often.

        Attachments

        1. RequestExecutorQueueFull.png
          150 kB
          Sumanth Pasupuleti
        2. heap dump showing each ImmediateFlusher taking upto 600MB.png
          321 kB
          Sumanth Pasupuleti
        3. BlockedEpollEventLoopFromThreadDump.png
          658 kB
          Sumanth Pasupuleti
        4. BlockedEpollEventLoopFromHeapDump.png
          186 kB
          Sumanth Pasupuleti

          Activity

            People

            • Assignee:
              sumanth.pasupuleti Sumanth Pasupuleti
              Reporter:
              sumanth.pasupuleti Sumanth Pasupuleti
              Authors:
              Sumanth Pasupuleti
              Reviewers:
              Benedict
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: