[CASSANDRA-19751] IllegalStateException when query on table having static columns during the Cassandra cluster upgrade from 3.11.4 to 4.0.11

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Platform: All
• Assignee: Unassigned
• Reporter: Alaykumar Barochia
• Created: 05/Jul/24 03:57
• Updated: 20/Aug/24 08:22
• Resolved: 10/Jul/24 00:06

Description

We are upgrading Cassandra cluster from 3.11.4 to 4.0.11. This cluster has SSL enabled.
While performing upgrade on 1st DC, we observed below WARN/ERROR messages on C* 3 and C* 4 nodes.

C*3 nodes:

WARN  [ReadStage-1] 2024-06-11 08:04:09,088 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {}
java.lang.IllegalStateException: [last_metadata_updt_ts, price_metadata] is not a subset of [price_metadata]

WARN  [ReadStage-1] 2024-06-19 05:10:31,226 AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread Thread[ReadStage-1,5,main]: {}
java.lang.IllegalStateException: [default_price_json, last_metadata_updt_ts, price_metadata] is not a subset of [price_metadata]

C*4 nodes:

ERROR [ReadStage-1] 2024-06-19 05:48:47,388 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[ReadStage-1,5,main]
java.lang.IllegalStateException: [last_metadata_updt_ts, price_metadata] is not a subset of [price_metadata]

Table definition for which above columns are associated is as below:

CREATE TABLE omni_price_ks_v2.location_price_mstr (
    tcin text,
    location_id bigint,
    price_change_id text,
    default_price_json text static,
    end_ts bigint,
    last_metadata_updt_ts bigint static,
    last_update_ts bigint,
    price_json text,
    price_metadata text static,
    price_type text,
    start_ts bigint,
    status text,
    version text,
    PRIMARY KEY (tcin, location_id, price_change_id)
) WITH CLUSTERING ORDER BY (location_id ASC, price_change_id ASC)
    AND bloom_filter_fp_chance = 0.1
    AND caching = {'keys': 'ALL', 'rows_per_partition': '100'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

App team also observed below error in their application logs when try to read from this table.

{ "code": "ERR_GETPRICE_0034", "message": "Cassandra failure during read query at consistency LOCAL_QUORUM (2 responses were required but only 1 replica responded, 1 failed)" }

Because of this error, the application is getting impacted during the upgrade.
Once the upgrade on all DCs is completed, this error stops.

I found below bug which matches our case.
https://issues.apache.org/jira/browse/CASSANDRA-17601

It seems like we are hitting some bug and hence raising this Jira.

Can you please have a look if this is still a bug and what would be the fix?

Let me know if you need any more details.

Issue Links

• duplicates: CASSANDRA-17601 IllegalStateException with prepared queries selecting static columns in mixed 3.0.x/4.x clusters