[CASSANDRA-16307] GROUP BY queries with paging can return deleted data

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.11.11, 4.0-rc1, 4.0
• Component/s: Consistency/Coordination
• Bug Category: Correctness - Consistency
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 3.10

Source Control Link

https://github.com/apache/cassandra/commit/7cddbd40ce6b326df533fd6d3c4131ef70b3b068

Test and Documentation Plan

Tests included. Harry tests are under way.

• Assignee: Alex Petrov
• Reporter: Andres de la Peña

Authors

Alex Petrov

Reviewers

Benjamin Lerer, Sam Tunnicliffe

• Created: 30/Nov/20 16:57
• Updated: 25/Feb/21 10:10
• Resolved: 15/Feb/21 11:56

Description

GROUP BY queries using paging and CL>ONE/LOCAL_ONE. This dtest reproduces the problem:

try (Cluster cluster = init(Cluster.create(2)))
{
    cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (pk int, ck int, PRIMARY KEY (pk, ck))"));
    ICoordinator coordinator = cluster.coordinator(1);
    coordinator.execute(withKeyspace("INSERT INTO %s.t (pk, ck) VALUES (0, 0)"), ConsistencyLevel.ALL);
    coordinator.execute(withKeyspace("INSERT INTO %s.t (pk, ck) VALUES (1, 1)"), ConsistencyLevel.ALL);
    
    cluster.get(1).executeInternal(withKeyspace("DELETE FROM %s.t WHERE pk=0 AND ck=0"));
    cluster.get(2).executeInternal(withKeyspace("DELETE FROM %s.t WHERE pk=1 AND ck=1"));
    String query = withKeyspace("SELECT * FROM %s.t GROUP BY pk");
    Iterator<Object[]> rows = coordinator.executeWithPaging(query, ConsistencyLevel.ALL, 1);
    assertRows(Iterators.toArray(rows, Object[].class));
}

Using a 2-node cluster and RF=2, the test inserts two partitions in both nodes. Then it locally deletes each row in a separate node, so each node sees a different partition alive, but reconciliation should produce no alive partitions. However, a GROUP BY query using a page size of 1 wrongly returns one of the rows.

This has been detected during CASSANDRA-16180, and it is probably related to CASSANDRA-15459, which solved a similar problem for group-by queries with limit, instead of paging.

Issue Links

• is blocked by: CASSANDRA-16427 In-JVM dtest paging does not handle Group By correctly
• is related to: CASSANDRA-15459 Short read protection doesn't work on group-by queries
• relates to: CASSANDRA-16180 4.0 Quality: Coordination Test Audit