[CASSANDRA-14475] nodetool - Occasional high CPU on large, CPU capable machines

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.3, 4.0-alpha1, 4.0
• Component/s: Tool/nodetool
• Severity: Normal
• Assignee: Thomas Steinmaurer
• Reporter: Thomas Steinmaurer

Authors

Thomas Steinmaurer

Reviewers

Jason Brown

• Created: 29/May/18 09:09
• Updated: 15/May/20 08:03
• Resolved: 29/May/18 12:22

Description

Periodically calling nodetool every 5 min results in increased CPU usage by nodetool only on a machine with 32 physical cores (64 vCPUs) according to our monitoring:

Investigation and testing has shown that it is related to running with default number of parallel GC threads which is 43 on this particular machine. We see a System.gc() according to flight recorder but no real evidence from where it comes from. The nodetool call in question is simply gathering e.g. the version with "nodetool version".

After explicitly setting the number of parallel GC threads to 1, the high CPU is entirely gone (see chart above), without impacting nodetool being executed successfully. 1 parallel GC thread should be sufficient for nodetool anyway I think.