[CASSANDRA-10433] Reduce contention in CompositeType instance interning

• Type: Improvement
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 2.2.4

Environment

Cassandra 2.2.1 running on 6 AWS c3.4xlarge nodes, CentOS 6.6

• Assignee: David Schlosnagle Assign to me
• Reporter: David Schlosnagle

Authors

David Schlosnagle

Reviewers

Sylvain Lebresne

• Created: 01/Oct/15 22:48
• Updated: 16/Apr/19 09:30
• Resolved: 30/Jun/16 15:14

Description

While running some workload tests on Cassandra 2.2.1 and profiling with flight recorder in a test environment, we have noticed significant contention on the static synchronized org.apache.cassandra.db.marshal.CompositeType.getInstance(List) method.

We are seeing threads blocked for 22.828 seconds from a 60 second snapshot while under a mix of reads and writes from a Thrift based client.

I would propose to reduce contention in org.apache.cassandra.db.marshal.CompositeType.getInstance(List) by using a ConcurrentHashMap for the instances cache.

Contention Back Trace
org.apache.cassandra.db.marshal.CompositeType.getInstance(List)
  org.apache.cassandra.db.composites.AbstractCompoundCellNameType.asAbstractType()
    org.apache.cassandra.db.SuperColumns.getComparatorFor(CFMetaData, boolean)
      org.apache.cassandra.db.SuperColumns.getComparatorFor(CFMetaData, ByteBuffer)
        org.apache.cassandra.thrift.ThriftValidation.validateColumnNames(CFMetaData, ByteBuffer, Iterable)
          org.apache.cassandra.thrift.ThriftValidation.validateColumnPath(CFMetaData, ColumnPath)
            org.apache.cassandra.thrift.ThriftValidation.validateColumnOrSuperColumn(CFMetaData, ByteBuffer, ColumnOrSuperColumn)
              org.apache.cassandra.thrift.ThriftValidation.validateMutation(CFMetaData, ByteBuffer, Mutation)
                org.apache.cassandra.thrift.CassandraServer.createMutationList(ConsistencyLevel, Map, boolean)
                  org.apache.cassandra.thrift.CassandraServer.batch_mutate(Map, ConsistencyLevel)
                    org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra$Iface, Cassandra$batch_mutate_args)
        org.apache.cassandra.thrift.ThriftValidation.validateRange(CFMetaData, ColumnParent, SliceRange)
          org.apache.cassandra.thrift.ThriftValidation.validatePredicate(CFMetaData, ColumnParent, SlicePredicate)
            org.apache.cassandra.thrift.CassandraServer.get_range_slices(ColumnParent, SlicePredicate, KeyRange, ConsistencyLevel)
              org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra$Iface, Cassandra$get_range_slices_args)
                org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Object, TBase)
                  org.apache.thrift.ProcessFunction.process(int, TProtocol, TProtocol, Object)
                    org.apache.thrift.TBaseProcessor.process(TProtocol, TProtocol)
                      org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run()
                        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
                          java.util.concurrent.ThreadPoolExecutor$Worker.run()
            org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(String, List, ColumnParent, long, SlicePredicate, ConsistencyLevel, ClientState)
              org.apache.cassandra.thrift.CassandraServer.multiget_slice(List, ColumnParent, SlicePredicate, ConsistencyLevel)
                org.apache.cassandra.thrift.Cassandra$Processor$multiget_slice.getResult(Cassandra$Iface, Cassandra$multiget_slice_args)
                  org.apache.cassandra.thrift.Cassandra$Processor$multiget_slice.getResult(Object, TBase)