[CASSANDRA-14958] Counters fail to increment in 2.1/2.2 to 3.X mixed version clusters

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 3.0.18, 3.11.4
• Component/s: Feature/Counters
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Critical
• Complexity: Challenging
• Discovered By: User Report
• Assignee: Aleksey Yeschenko
• Reporter: Ariel Weisberg

Authors

Aleksey Yeschenko

Reviewers

Benedict Elliott Smith

• Created: 07/Jan/19 16:15
• Updated: 02/Aug/19 02:40
• Resolved: 14/Jan/19 18:09

Description

The upgrade test for this is failing
https://circleci.com/gh/aweisberg/cassandra/2362#tests/containers/1

I confirmed that this is occurring manually using cqlsh against the cluster constructed by the dtest.

cqlsh> describe schema;

CREATE KEYSPACE ks WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;

CREATE TABLE ks.clicks (
    userid int,
    url text,
    total counter,
    PRIMARY KEY (userid, url)
) WITH COMPACT STORAGE
    AND CLUSTERING ORDER BY (url ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

cqlsh> use ks;
cqlsh:ks> UPDATE clicks SET total = total + 1 WHERE userid = 1 AND url = 'http://foo.com';
cqlsh:ks> SELECT total FROM clicks WHERE userid = 1 AND url = 'http://foo.com'
      ... ;

 total
-------
     0

(1 rows)

Issue Links

• is caused by: CASSANDRA-13691 Fix incorrect [2.1 <— 3.0] serialization of counter cells with pre-2.1 local shards
• is duplicated by: CASSANDRA-14881 Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 3.11.2