Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14318

Fix query pager DEBUG log leak causing hit in paged reads throughput

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.2.13
    • None
    • Normal

    Description

      Debug logging can involve in many cases (especially very low latency ones) a very important overhead on the read path in 2.2 as we've seen when upgrading clusters from 2.0 to 2.2.

      The performance impact was especially noticeable on the client side metrics, where p99 could go up to 10 times higher, while ClientRequest metrics recorded by Cassandra didn't show any overhead.

      Below shows latencies recorded on the client side with debug logging on first, and then without it :

       

      We generated a flame graph before turning off debug logging that shows the read call stack is dominated by debug logging : 

      I've attached the original flame graph for exploration.

      Once disabled, the new flame graph shows that the read call stack gets extremely thin, which is further confirmed by client recorded metrics : 

      The query pager code has been reworked since 3.0 and it looks like log.debug() calls are gone there, but for 2.2 users and to prevent such issues to appear with default settings, I really think debug logging should be disabled by default.

      Attachments

        1. cassandra-2.2-debug.yaml
          1 kB
          Alexander Dejanovski
        2. debuglogging.png
          57 kB
          Alexander Dejanovski
        3. flame_graph_snapshot.png
          208 kB
          Alexander Dejanovski
        4. flame22-nodebug-sjk.svg
          51 kB
          Alexander Dejanovski
        5. flame22 nodebug sjk svg.png
          105 kB
          Alexander Dejanovski
        6. flame22-sjk.svg
          76 kB
          Alexander Dejanovski

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            adejanovski Alexander Dejanovski Assign to me
            adejanovski Alexander Dejanovski
            Alexander Dejanovski
            Paulo Motta
            Votes:
            2 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment