[CASSANDRA-9272] Atomic Batch Statements Fail to Execute on Some Coordinators

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem

Environment

Operating System: Red Hat Enterprise Linux Server release 6.5 (Santiago) |
Cassandra and CQL: [cqlsh 4.1.1 | Cassandra 2.0.14 | CQL spec 3.1.1 | Thrift protocol 19.39.0] |
JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_79 (Oracle Java) |
Using PropertyFileSnitch to define cluster topology.

• Severity: Normal
• Since Version: 2.0.6
• Assignee: Carl Yeksigian
• Reporter: Jonathan Gray

Authors

Carl Yeksigian

• Created: 30/Apr/15 18:53
• Updated: 16/Apr/19 09:31
• Resolved: 04/May/15 21:57

Description

Install and configure a three-node Cassandra cluster spread across two data centers (DCs). Place two Cassandra instances in DC1 and one Cassandra instance in DC2. (We have designated one node as a seed in each data center.)

Gracefully shut down any one Cassandra instance in DC1. This can be done by running 'kill <process_id>' in Linux.

Open a CQL Shell connection to the remaining node in DC1.

Create the following keyspace: CREATE KEYSPACE batch_test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2':'1'};

Create the following table: CREATE TABLE batch_test.batch_table ( partition_key TEXT PRIMARY KEY );

Execute the following atomic batch query: BEGIN BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;

Observe an availability-related error, despite a sufficient number of nodes available to service the query. The trace below shows sample results from our cluster:

DC1

---

cqlsh> CONSISTENCY
Current consistency level is ONE.
cqlsh> BEGIN BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;
Unable to complete request: one or more nodes were unavailable.
cqlsh> CONSISTENCY QUORUM
Consistency level set to QUORUM.
cqlsh> BEGIN BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;
Unable to complete request: one or more nodes were unavailable.
cqlsh> BEGIN UNLOGGED BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;
cqlsh>

Note: an UNLOGGED BATCH executes successfully. A regular BATCH also executes successfully if it only contains one INSERT query.

We now show a similar trace using a CQL Shell connected to the Cassandra instance in DC2:

DC2
—

cqlsh> CONSISTENCY
Current consistency level is ONE.
cqlsh> BEGIN BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;
cqlsh> CONSISTENCY QUORUM
Consistency level set to QUORUM.
cqlsh> BEGIN BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;
cqlsh> BEGIN UNLOGGED BATCH INSERT INTO batch_test.batch_table (partition_key) VALUES ('a'); INSERT INTO batch_test.batch_table (partition_key) VALUES ('b'); APPLY BATCH;
cqlsh>

All queries execute successfully using the DC2 Cassandra instance as a coordinator.