Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13756

StreamingHistogram is not thread safe

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 3.0.15, 3.11.1
    • None
    • None
    • Normal

    Description

      When we test C*3 in shadow cluster, we notice after a period of time, several data node suddenly run into 100% cpu and stop process query anymore.

      After investigation, we found that threads are stuck on the sum() in streaminghistogram class. Those are jmx threads that working on expose getTombStoneRatio metrics (since jmx is kicked off every 3 seconds, there is a chance that multiple jmx thread is access streaminghistogram at the same time).

      After further investigation, we find that the optimization in CASSANDRA-13038 led to a spool flush every time when we call sum(). Since TreeMap is not thread safe, threads will be stuck when multiple threads visit sum() at the same time.

      There are two approaches to solve this issue.

      The first one is to add a lock to the flush in sum() which will introduce some extra overhead to streaminghistogram.

      The second one is to avoid streaminghistogram to be access by multiple threads. For our specific case, is to remove the metrics we added.

      Attachments

        Issue Links

          Activity

            People

              jjirsa Jeff Jirsa
              xiaxiangzhou xiangzhou xia
              Jeff Jirsa
              Jason Brown
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: