Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7758

Some gossip messages are very slow to process on vnode clusters

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.2.19, 2.0.10
    • Component/s: None
    • Labels:
      None

      Description

      Both StorageService.getChangedRangesForLeaving and PendingRangeCalculatorService.calculatePendingRanges clone the TokenMetadata map once per range that a node covers. This is a very expensive operation on a cluster with vnodes since the TokenMetadata map can easily be tens of thousands of entries, even for a small cluster, and this runs thousands of times for each changing node. This can cause big cluster-wide slowdowns & problems in the GossipStage and PendingRangeCalculator thread pools. Processing just one message can generate enough garbage to pause the entire cluster simultaneously when one of these is being propagated. Patches forthcoming.

        Attachments

          Activity

            People

            • Assignee:
              rbranson Rick Branson Assign to me
              Reporter:
              rbranson Rick Branson
              Authors:
              Rick Branson
              Reviewers:
              Jonathan Ellis

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment