Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-19107

Revert unnecessary read lock acquisition when reading ring version in TokenMetadata introduced in CASSANDRA-16286

    XMLWordPrintableJSON

Details

    Description

      CASSANDRA-16286 achieved its goal of making sure that concurrent increments to the ring version would independently increment the version (i.e. not "merge" multiple invalidations into single versions), but it also unnecessarily replaced the volatile read on ringVersion w/ making readVersion non-volatile and acquiring the read lock on the fair ReadWriteLock in TokenMetadata. This can result in unnecessary queueing w/ high CPU usage/read volume. For example, you might see this on a 4.0 cluster...

      "Native-Transport-Requests-99" #271 daemon prio=5 os_prio=0 cpu=5822566.56ms elapsed=19477779.40s tid=0x00007fcc96c31b00 nid=0xb7bd waiting on condition  [0x00007fcb7f144000]
        java.lang.Thread.State: WAITING (parking)
             at jdk.internal.misc.Unsafe.park(java.base@11.0.16/Native Method)
             - parking to wait for  <0x00000005c0ab92a0> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
             at java.util.concurrent.locks.LockSupport.park(java.base@11.0.16/LockSupport.java:194)
             at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.16/AbstractQueuedSynchronizer.java:885)
             at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.16/AbstractQueuedSynchronizer.java:1009)
             at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.16/AbstractQueuedSynchronizer.java:1324)
             at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.16/ReentrantReadWriteLock.java:738)
             at org.apache.cassandra.locator.TokenMetadata.getRingVersion(TokenMetadata.java:1389)
             at org.apache.cassandra.locator.AbstractReplicationStrategy.getCachedReplicas(AbstractReplicationStrategy.java:82)
             at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicas(AbstractReplicationStrategy.java:116)
             at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalReplicasForToken(AbstractReplicationStrategy.java:109)
             at org.apache.cassandra.locator.ReplicaLayout.forTokenWriteLiveAndDown(ReplicaLayout.java:209)
             at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:328)
             at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1426)
             at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:937)
      

      Reverting to a volatile read makes this no longer possible, but keeps the fix from CASSANDRA-16286 intact.

      Attachments

        1. result_details_50.tar.gz
          37.02 MB
          Caleb Rackliffe
        2. ci_summary_50.html
          7 kB
          Caleb Rackliffe
        3. result_details.tar-1.gz
          24.86 MB
          Caleb Rackliffe
        4. ci_summary-1.html
          7 kB
          Caleb Rackliffe
        5. result_details.tar.gz
          16.78 MB
          Caleb Rackliffe
        6. ci_summary.html
          7 kB
          Caleb Rackliffe

        Issue Links

          Activity

            People

              maedhroz Caleb Rackliffe
              maedhroz Caleb Rackliffe
              Caleb Rackliffe
              Francisco Guerrero
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h