[CASSANDRA-11087] Queries on compact storage tables in mixed version clusters can return incorrect results

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.3, 3.3
• Component/s: Legacy/Coordination
• Severity: Normal
• Since Version: 3.0.0
• Assignee: Sam Tunnicliffe
• Reporter: Sam Tunnicliffe

Authors

Sam Tunnicliffe

• Created: 28/Jan/16 14:04
• Updated: 16/Apr/19 09:30
• Resolved: 02/Feb/16 14:50

Description

Whilst writing a dtest for CASSANDRA-11045, it becomes apparent that queries on compact storage tables are broken during the 3.0 upgrade (and this has probably been the case since day 1).

tl;dr In a cluster with a mix of < 3.0 and 3.0 nodes, reads on COMPACT STORAGE tables may not include all results.

To repro: tables are created and data written before any nodes are upgraded to 3.0+, some nodes are then upgraded putting the cluster into a mixed state.
Now, when a query is run where the coordinator is a < 3.0 node, any 3.0+ replica which has not yet run upgradesstables always returns 0 results. Once upgradesstables is run, the replica returns the correct results. Likewise, if the data is inserted after the node is upgraded, the results are correct. If the 3.0 node acts as the coordinator, the results are also correct and so once all nodes are upgraded, the problem goes away.

The behaviour can be seen for both single partition and range requests as this dtest demonstrates.