Details
-
Bug
-
Status: Resolved
-
Urgent
-
Resolution: Duplicate
-
None
-
None
-
None
-
Critical
Description
MessagingSerice is a system singleton with a static Multimap field targets. Multimaps are not thread safe but no attempt is made to synchronize access to that field. Multimap ultimately uses the standard java HashMap which is susceptible to a race condition where threads will get stuck during a get operation yielding multiple threads similar to the following stack:
"pool-1-thread-6451" prio=10 tid=0x00007fa5242c9000 nid=0x10f4 runnable [0x00007fa52fde4000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at com.google.common.collect.AbstractMultimap.getOrCreateCollection(AbstractMultimap.java:205)
at com.google.common.collect.AbstractMultimap.put(AbstractMultimap.java:194)
at com.google.common.collect.AbstractListMultimap.put(AbstractListMultimap.java:72)
at com.google.common.collect.ArrayListMultimap.put(ArrayListMultimap.java:60)
at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:303)
at org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:353)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:229)
at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:98)
at org.apache.cassandra.thrift.CassandraServer.get(CassandraServer.java:289)
at org.apache.cassandra.thrift.Cassandra$Processor$get.process(Cassandra.java:2655)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)