[CASSANDRA-16334] Replica failure causes timeout on multi-DC write

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.26, 3.11.12, 4.0.2, 4.1-alpha1, 4.1
• Component/s: Consistency/Coordination, Messaging/Internode
• Bug Category: Correctness - API / Semantic Implementation
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra/commit/7f54fe02298b90e6152acc026384c033a96ce621

Test and Documentation Plan

Added a dtest that covers both bugs causing replica failures to appear as timeouts.

• Assignee: Alex Sorokoumov
• Reporter: Paulo Motta

Authors

Alex Sorokoumov

Reviewers

Andres de la Peña, Paulo Motta

• Created: 11/Dec/20 11:49
• Updated: 27/May/22 19:25
• Resolved: 26/Oct/21 17:33

Description

Inserting a mutation larger than max_mutation_size_in_kb correctly throws a write error on a single DC keyspace with RF=3:

cassandra.WriteFailure: Error from server: code=1500 [Replica(s) failed to execute write] message="Operation failed - received 0 responses and 3 failures: UNKNOWN from /127.0.0.3:7000, UNKNOWN from /127.0.0.2:7000, UNKNOWN from /127.0.0.1:7000" info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0, 'failures': 3}

The same insert wrongly causes a timeout on a keyspace with 2 dcs (RF=3 each):

cassandra.WriteTimeout: Error from server: code=1100 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'consistency': 'LOCAL_ONE', 'required_responses': 1, 'received_responses': 0}

Reproduction steps:

# Setup cluster
ccm create -n 3:3 test
for i in {1..6}; do echo 'max_mutation_size_in_kb: 1000' >> ~/.ccm/test/node$i/conf/cassandra.yaml; done
ccm start

# Create schema
ccm node1 cqlsh
CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};
CREATE TABLE test.test (key int PRIMARY KEY, val blob);
exit;

# Insert data
python
from cassandra.cluster import Cluster
cluster = Cluster()
session = cluster.connect('test')
blob = f = open("2mbBlob", "rb").read().hex()
session.execute("INSERT INTO test (key, val) VALUES (1, textAsBlob('" + blob + "'))")

Reproduced in 3.0, 3.11, 4.0, trunk.

Issue Links

• Testing discovered: CASSANDRA-17043 CircleCI dtest multiplexer with MIDRES needs more resources