[CASSANDRA-10041] "timeout during write query at consistency ONE" when updating counter at consistency QUORUM and 2 of 3 nodes alive

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem
• Fix Version/s: 2.1.x

Environment

centos 6.6 server, java version "1.8.0_45", cassandra 2.1.8, 3 machines, keyspace with replication factor 3

• Severity: Normal
• Assignee: Unassigned
• Reporter: Anton Lebedevich
• Created: 11/Aug/15 09:40
• Updated: 16/Apr/19 09:31
• Resolved: 07/Apr/16 08:58

Description

Test scenario is: kill -9 one node, wait 60 seconds, start it back, wait till it becomes available, wait 120 seconds (during that time all 3 nodes are up), repeat with the next node. Application reads from one table and updates counters in another table with consistency QUORUM. When one node out of 3 is killed application logs this exception for several seconds:

Caused by: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency ONE (1 replica were required but only 0 acknowledged the write)
        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:57) ~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
        at com.datastax.driver.core.Responses$Error$1.decode(Responses.java:37) ~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
        at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:204) ~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
        at com.datastax.driver.core.Message$ProtocolDecoder.decode(Message.java:195) ~[com.datastax.cassandra.cassandra-driver-core-2.1.6.jar:na]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) [io.netty.netty-codec-4.0.27.Final.jar:4.0.27.Final]
        ... 13 common frames omitted