Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14475

nodetool - Occasional high CPU on large, CPU capable machines

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 3.11.3, 4.0
    • Component/s: Tool/nodetool
    • Labels:
      None
    • Severity:
      Normal

      Description

      Periodically calling nodetool every 5 min results in increased CPU usage by nodetool only on a machine with 32 physical cores (64 vCPUs) according to our monitoring:

      Investigation and testing has shown that it is related to running with default number of parallel GC threads which is 43 on this particular machine. We see a System.gc() according to flight recorder but no real evidence from where it comes from. The nodetool call in question is simply gathering e.g. the version with "nodetool version".

      After explicitly setting the number of parallel GC threads to 1, the high CPU is entirely gone (see chart above), without impacting nodetool being executed successfully. 1 parallel GC thread should be sufficient for nodetool anyway I think.

        Attachments

        1. nodetool_highcpu_gcthreads1_cassandra_JIRA.png
          42 kB
          Thomas Steinmaurer
        2. nodetool_gc_threads.patch
          0.4 kB
          Thomas Steinmaurer

          Activity

            People

            • Assignee:
              tsteinmaurer Thomas Steinmaurer
              Reporter:
              tsteinmaurer Thomas Steinmaurer
              Authors:
              Thomas Steinmaurer
              Reviewers:
              Jason Brown
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: