[CASSANDRA-14881] Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 3.11.2

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Legacy/Core

Environment

// code placeholder

• Severity: Normal
• Since Version: 2.1.16
• Assignee: Unassigned
• Reporter: Laxmikant Upadhyay
• Created: 09/Nov/18 06:54
• Updated: 16/Apr/19 09:29
• Resolved: 14/Jan/19 18:11

Description

I have 3 node (A,B,C) 2.1.16 cassandra cluster which i am upgrading to cassandra 3.11.2.
My current cluster status is node a has been upgrade to 3.11.2, B is down, and C is on cassandra 2.1.16

when i run counter update using cqlsh it is behaving strange inconsistent way , sometimes the update actually applied and sometimes it does not apply at all.

See the below example of cqlsh logged in to node A:
===============Incorrect update : update not applied ========

user@cqlsh:ks> select * from "counterTable";

 key              | column1        | value
------------------+----------------+-------
 key1 | count |     0
 
user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 'key1' AND column1 = 'count';

user@cqlsh:ks> select * from "counterTable";

 key              | column1        | value
------------------+----------------+-------
 key1 | count |     0

 ===============Correct update : update applied successfully ========

user@cqlsh> USE ks ;
user@cqlsh:ks> select * from "counterTable";

 key              | column1        | value
------------------+----------------+-------
 key1 | count |  -100
 
user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 'key1' AND column1 = 'count';

user@cqlsh:ks> select * from "counterTable";

 key              | column1        | value
------------------+----------------+-------
 key1 | count |     0

I have attached the output with trace enabled (with actual timestamp) for both correct and incorrect counter update .

What is the reason of this weird behavior ?

Below is my table details:

CREATE TABLE ks."counterTable" (
    key text,
    column1 text,
    value counter,
    PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
    AND CLUSTERING ORDER BY (column1 ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Note : When all nodes are successfully upgraded OR when 2 nodes gets upgraded to 3.11.2 and 3rd non-upgraded node is down then we don't see such issue . couner update works as expected

Issue Links

• duplicates: CASSANDRA-14958 Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters