Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15013

Prevent client requests from blocking on executor task queue

    XMLWordPrintableJSON

    Details

      Description

      This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue bounded, since, in the current state, items get added to the queue without any checks on queue size, nor with any checks on netty outbound buffer to check the isWritable state.
      We are seeing this issue hit our production 3.0 clusters quite often.

        Attachments

        1. 15013-trunk.txt
          55 kB
          Sumanth Pasupuleti
        2. 15013-3.0.txt
          58 kB
          Sumanth Pasupuleti
        3. 15013-3.11.txt
          58 kB
          Sumanth Pasupuleti
        4. perftest2_15013_base_flamegraph.svg
          622 kB
          Sumanth Pasupuleti
        5. perftest2_15013_patch_flamegraph.svg
          741 kB
          Sumanth Pasupuleti
        6. perftest2_heap.png
          1.20 MB
          Sumanth Pasupuleti
        7. perftest2_cpu_usage.png
          200 kB
          Sumanth Pasupuleti
        8. perftest2_blocked_threadpool.png
          89 kB
          Sumanth Pasupuleti
        9. perftest2_write_latency_99th.png
          346 kB
          Sumanth Pasupuleti
        10. perftest2_write_latency_avg.png
          252 kB
          Sumanth Pasupuleti
        11. perftest2_writeops.png
          263 kB
          Sumanth Pasupuleti
        12. perftest2_read_latency_avg.png
          258 kB
          Sumanth Pasupuleti
        13. perftest2_read_latency_99th.png
          340 kB
          Sumanth Pasupuleti
        14. perftest2_readops.png
          275 kB
          Sumanth Pasupuleti
        15. perftest_connections_count.png
          259 kB
          Sumanth Pasupuleti
        16. perftest_heap_usage.png
          513 kB
          Sumanth Pasupuleti
        17. perftest_cpu_usage.png
          282 kB
          Sumanth Pasupuleti
        18. perftest_writelatency_avg.png
          216 kB
          Sumanth Pasupuleti
        19. perftest_writelatency_99th.png
          280 kB
          Sumanth Pasupuleti
        20. perftest_readlatency_avg.png
          208 kB
          Sumanth Pasupuleti
        21. perftest_readlatency_99th.png
          246 kB
          Sumanth Pasupuleti
        22. perftest_readops.png
          213 kB
          Sumanth Pasupuleti
        23. perftest_blockedthreads.png
          176 kB
          Sumanth Pasupuleti
        24. perftest_writeops.png
          213 kB
          Sumanth Pasupuleti
        25. RequestExecutorQueueFull.png
          150 kB
          Sumanth Pasupuleti
        26. BlockedEpollEventLoopFromThreadDump.png
          658 kB
          Sumanth Pasupuleti
        27. BlockedEpollEventLoopFromHeapDump.png
          186 kB
          Sumanth Pasupuleti
        28. heap dump showing each ImmediateFlusher taking upto 600MB.png
          321 kB
          Sumanth Pasupuleti

          Activity

            People

            • Assignee:
              sumanth.pasupuleti Sumanth Pasupuleti
              Reporter:
              sumanth.pasupuleti Sumanth Pasupuleti
              Authors:
              Sumanth Pasupuleti
              Reviewers:
              Benedict Elliott Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: