Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7758

Some gossip messages are very slow to process on vnode clusters

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 1.2.19, 2.0.10
    • None
    • None

    Description

      Both StorageService.getChangedRangesForLeaving and PendingRangeCalculatorService.calculatePendingRanges clone the TokenMetadata map once per range that a node covers. This is a very expensive operation on a cluster with vnodes since the TokenMetadata map can easily be tens of thousands of entries, even for a small cluster, and this runs thousands of times for each changing node. This can cause big cluster-wide slowdowns & problems in the GossipStage and PendingRangeCalculator thread pools. Processing just one message can generate enough garbage to pause the entire cluster simultaneously when one of these is being propagated. Patches forthcoming.

      Attachments

        1. 7758-1-v2.0.txt
          2 kB
          Rick Branson
        2. 7758-1-v1.2.txt
          2 kB
          Rick Branson

        Activity

          People

            rbranson Rick Branson
            rbranson Rick Branson
            Rick Branson
            Jonathan Ellis
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: