[CASSANDRA-17047] Dropping a column can break queries until the schema is fully propagated (TAKE 2)

• Type: Bug
• Status: Patch Available
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.0.x, 3.11.x, 4.0.x
• Component/s: Cluster/Schema
• Bug Category: Correctness - Transient Incorrect Response
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All

Test and Documentation Plan

The patch add additional in-jvm DTest tests

• Assignee: Benjamin Lerer
• Reporter: Benjamin Lerer

Authors

Benjamin Lerer

Reviewers

Berenguer Blasi, Jacek Lewandowski

• Created: 18/Oct/21 17:05
• Updated: 08/Jun/23 04:52

Description

With a table like:

CREATE TABLE ks.tbl (id int primary key, v1 int, v2 int, v3 int)

and we drop v2, we get this exception on the replicas which haven't seen the schema change:

ERROR [SharedPool-Worker-1] node2 2020-06-24 09:49:08,107 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,node2]
java.lang.IllegalStateException: [ColumnDefinition{name=v1, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, position=-1}, ColumnDefinition{name=v2, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, position=-1}, ColumnDefinition{name=v3, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, position=-1}] is not a subset of [v1 v3]
	at org.apache.cassandra.db.Columns$Serializer.encodeBitmap(Columns.java:546) ~[main/:na]
	at org.apache.cassandra.db.Columns$Serializer.serializeSubset(Columns.java:478) ~[main/:na]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:184) ~[main/:na]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:114) ~[main/:na]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.serialize(UnfilteredSerializer.java:102) ~[main/:na]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:132) ~[main/:na]
	at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) ~[main/:na]
...

Note that it doesn't matter if we SELECT * or SELECT id, v1

CASSANDRA-15899 tried to fix the problem when columns are dropped as well as when columns are added. Unfortunately the fix introduced an issue and had to be reverted in CASSANDRA-16735.

If the scenario for ADDED columns is tricky, the original scenario for DROPPED columns can be solved in a safe way at the ColumnFilter level. By consequence, I think that we should at least solve that scenario.

bdeggleston, samt, ifesdjeen does my proposal makes sense to you?

Issue Links

• relates to: CASSANDRA-18532 Cannot read or repair data after dropping a column when one of the nodes is down
• links to: GitHub Pull Request #1286
• links to: GitHub Pull Request #2392