Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15013

Prevent client requests from blocking on executor task queue

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      This is a follow-up ticket out of CASSANDRA-14855, to make the Flusher queue bounded, since, in the current state, items get added to the queue without any checks on queue size, nor with any checks on netty outbound buffer to check the isWritable state.
      We are seeing this issue hit our production 3.0 clusters quite often.

      Attachments

        1. 15013-trunk.txt
          55 kB
          Sumanth Pasupuleti
        2. 15013-3.0.txt
          58 kB
          Sumanth Pasupuleti
        3. 15013-3.11.txt
          58 kB
          Sumanth Pasupuleti
        4. perftest2_15013_base_flamegraph.svg
          622 kB
          Sumanth Pasupuleti
        5. perftest2_15013_patch_flamegraph.svg
          741 kB
          Sumanth Pasupuleti
        6. perftest2_heap.png
          1.20 MB
          Sumanth Pasupuleti
        7. perftest2_cpu_usage.png
          200 kB
          Sumanth Pasupuleti
        8. perftest2_blocked_threadpool.png
          89 kB
          Sumanth Pasupuleti
        9. perftest2_write_latency_99th.png
          346 kB
          Sumanth Pasupuleti
        10. perftest2_write_latency_avg.png
          252 kB
          Sumanth Pasupuleti
        11. perftest2_writeops.png
          263 kB
          Sumanth Pasupuleti
        12. perftest2_read_latency_avg.png
          258 kB
          Sumanth Pasupuleti
        13. perftest2_read_latency_99th.png
          340 kB
          Sumanth Pasupuleti
        14. perftest2_readops.png
          275 kB
          Sumanth Pasupuleti
        15. perftest_connections_count.png
          259 kB
          Sumanth Pasupuleti
        16. perftest_heap_usage.png
          513 kB
          Sumanth Pasupuleti
        17. perftest_cpu_usage.png
          282 kB
          Sumanth Pasupuleti
        18. perftest_writelatency_avg.png
          216 kB
          Sumanth Pasupuleti
        19. perftest_writelatency_99th.png
          280 kB
          Sumanth Pasupuleti
        20. perftest_readlatency_avg.png
          208 kB
          Sumanth Pasupuleti
        21. perftest_readlatency_99th.png
          246 kB
          Sumanth Pasupuleti
        22. perftest_readops.png
          213 kB
          Sumanth Pasupuleti
        23. perftest_blockedthreads.png
          176 kB
          Sumanth Pasupuleti
        24. perftest_writeops.png
          213 kB
          Sumanth Pasupuleti
        25. RequestExecutorQueueFull.png
          150 kB
          Sumanth Pasupuleti
        26. BlockedEpollEventLoopFromThreadDump.png
          658 kB
          Sumanth Pasupuleti
        27. BlockedEpollEventLoopFromHeapDump.png
          186 kB
          Sumanth Pasupuleti
        28. heap dump showing each ImmediateFlusher taking upto 600MB.png
          321 kB
          Sumanth Pasupuleti

        Issue Links

        is related to

        Task - A task that needs to be done. CASSANDRA-16100 Blog post on Improving Resiliency w/ "Prevent client requests from blocking on executor task queue"

        • Normal -
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sumanth.pasupuleti Sumanth Pasupuleti Assign to me
            sumanth.pasupuleti Sumanth Pasupuleti
            Sumanth Pasupuleti
            Benedict Elliott Smith
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment