[CASSANDRA-13043] UnavailabeException caused by counter writes forwarded to leaders without complete cluster view

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 3.0.15, 3.11.1
• Component/s: Legacy/Coordination

Environment

Debian

• Severity: Low
• Assignee: Stefano Ortolani
• Reporter: Catalin Alexandru Zamfir

Authors

Stefano Ortolani

Reviewers

Aleksey Yeschenko

• Created: 14/Dec/16 10:43
• Updated: 16/Apr/19 09:30
• Resolved: 20/Sep/17 19:29

Description

In version 3.9 of Cassandra, we get the following exceptions on the system.log whenever booting an agent. They seem to grow in number with each reboot. Any idea where they come from or what can we do about them? Note that the cluster is healthy (has sufficient live nodes).

2/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
12/14/2016 12:39:47 PMINFO  10:39:47 Updating topology for /10.136.64.120
12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-111,5,main]: {}
12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-118,5,main]: {}
12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-164,5,main]: {}
12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
12/14/2016 12:39:47 PMWARN  10:39:47 Uncaught exception on thread Thread[CounterMutationStage-117,5,main]: {}
12/14/2016 12:39:47 PMorg.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level LOCAL_QUORUM
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.ConsistencyLevel.assureSufficientLiveNodes(ConsistencyLevel.java:313) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.AbstractWriteResponseHandler.assureSufficientLiveNodes(AbstractWriteResponseHandler.java:146) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:1054) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:1450) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:48) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.9.jar:3.9]
12/14/2016 12:39:47 PM	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

Issue Links

• is related to: CASSANDRA-14028 counter_leader_with_partial_view_test-novnodes - counter_tests.TestCounters fails
• is related to: CASSANDRA-13914 Advertise RPC_READY for both native protocol and Thrift
• links to: GitHub Pull Request #224