Cassandra
  1. Cassandra
  2. CASSANDRA-6345

Endpoint cache invalidation causes CPU spike (on vnode rings?)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.2.13, 2.0.4
    • Component/s: None
    • Labels:
      None
    • Environment:

      30 nodes total, 2 DCs
      Cassandra 1.2.11
      vnodes enabled (256 per node)

      Description

      We've observed that events which cause invalidation of the endpoint cache (update keyspace, add/remove nodes, etc) in AbstractReplicationStrategy result in several seconds of thundering herd behavior on the entire cluster.

      A thread dump shows over a hundred threads (I stopped counting at that point) with a backtrace like this:

      at java.net.Inet4Address.getAddress(Inet4Address.java:288)
      at org.apache.cassandra.locator.TokenMetadata$1.compare(TokenMetadata.java:106)
      at org.apache.cassandra.locator.TokenMetadata$1.compare(TokenMetadata.java:103)
      at java.util.TreeMap.getEntryUsingComparator(TreeMap.java:351)
      at java.util.TreeMap.getEntry(TreeMap.java:322)
      at java.util.TreeMap.get(TreeMap.java:255)
      at com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:200)
      at com.google.common.collect.AbstractSetMultimap.put(AbstractSetMultimap.java:117)
      at com.google.common.collect.TreeMultimap.put(TreeMultimap.java:74)
      at com.google.common.collect.AbstractMultimap.putAll(AbstractMultimap.java:273)
      at com.google.common.collect.TreeMultimap.putAll(TreeMultimap.java:74)
      at org.apache.cassandra.utils.SortedBiMultiValMap.create(SortedBiMultiValMap.java:60)
      at org.apache.cassandra.locator.TokenMetadata.cloneOnlyTokenMap(TokenMetadata.java:598)
      at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:104)
      at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:2671)
      at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:375)

      It looks like there's a large amount of cost in the TokenMetadata.cloneOnlyTokenMap that AbstractReplicationStrategy.getNaturalEndpoints is calling each time there is a cache miss for an endpoint. It seems as if this would only impact clusters with large numbers of tokens, so it's probably a vnodes-only issue.

      Proposal: In AbstractReplicationStrategy.getNaturalEndpoints(), cache the cloned TokenMetadata instance returned by TokenMetadata.cloneOnlyTokenMap(), wrapping it with a lock to prevent stampedes, and clearing it in clearEndpointCache(). Thoughts?

      1. half-way-thru-6345-rbranson-patch-applied.png
        31 kB
        Rick Branson
      2. 6345-v5.txt
        8 kB
        Jonathan Ellis
      3. 6345-v4.txt
        9 kB
        Jonathan Ellis
      4. 6345-v3.txt
        4 kB
        Jonathan Ellis
      5. 6345-v2.txt
        3 kB
        Jonathan Ellis
      6. 6345-rbranson-v2.txt
        10 kB
        Rick Branson
      7. 6345-rbranson.txt
        5 kB
        Rick Branson
      8. 6345.txt
        3 kB
        Jonathan Ellis

        Issue Links

          Activity

          Rick Branson created issue -
          Rick Branson made changes -
          Field Original Value New Value
          Reproduced In 1.2.11 [ 12325022 ]
          Jonathan Ellis made changes -
          Attachment 6345.txt [ 12613688 ]
          Jonathan Ellis made changes -
          Attachment 6345-v2.txt [ 12613697 ]
          Jonathan Ellis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Jonathan Ellis [ jbellis ]
          Reviewer Rick Branson [ rbranson ]
          Fix Version/s 1.2.12 [ 12325272 ]
          Fix Version/s 2.0.3 [ 12325322 ]
          Rick Branson made changes -
          Attachment 6345.txt [ 12613703 ]
          Rick Branson made changes -
          Rick Branson made changes -
          Attachment 6345.txt [ 12613703 ]
          Rick Branson made changes -
          Attachment 6345-rbranson.txt [ 12613706 ]
          Rick Branson made changes -
          Attachment 6345-rbranson-v2.txt [ 12614030 ]
          Jonathan Ellis made changes -
          Attachment 6345-v3.txt [ 12614329 ]
          Jonathan Ellis made changes -
          Attachment 6345-v4.txt [ 12614786 ]
          Sylvain Lebresne made changes -
          Fix Version/s 1.2.13 [ 12325602 ]
          Fix Version/s 1.2.12 [ 12325272 ]
          Fix Version/s 2.0.3 [ 12325322 ]
          Jonathan Ellis made changes -
          Attachment 6345-v5.txt [ 12615340 ]
          Jonathan Ellis made changes -
          Link This issue is related to CASSANDRA-6127 [ CASSANDRA-6127 ]
          Jonathan Ellis made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.0.4 [ 12325644 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Jonathan Ellis
              Reporter:
              Rick Branson
              Reviewer:
              Rick Branson
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development