[CASSANDRA-7758] Some gossip messages are very slow to process on vnode clusters

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.2.19, 2.0.10
• Severity: Normal
• Since Version: 1.2.0 beta 1
• Assignee: Rick Branson
• Reporter: Rick Branson

Authors

Rick Branson

Reviewers

Jonathan Ellis

• Created: 12/Aug/14 18:19
• Updated: 16/Apr/19 09:31
• Resolved: 15/Aug/14 19:39

Description

Both StorageService.getChangedRangesForLeaving and PendingRangeCalculatorService.calculatePendingRanges clone the TokenMetadata map once per range that a node covers. This is a very expensive operation on a cluster with vnodes since the TokenMetadata map can easily be tens of thousands of entries, even for a small cluster, and this runs thousands of times for each changing node. This can cause big cluster-wide slowdowns & problems in the GossipStage and PendingRangeCalculator thread pools. Processing just one message can generate enough garbage to pause the entire cluster simultaneously when one of these is being propagated. Patches forthcoming.