[CASSANDRA-18532] Cannot read or repair data after dropping a column when one of the nodes is down

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Cluster/Schema
• Bug Category: Availability - Unavailable
• Severity: Normal
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Assignee: Jakub Zytka
• Reporter: Jacek Lewandowski

Authors

Jakub Zytka

• Created: 17/May/23 08:54
• Updated: 05/Jun/23 14:51
• Resolved: 05/Jun/23 14:51

Description

When a column is dropped while one of the nodes is down, when the node come back online, repair or querying the data fails.

This can be reproduced by running the following in-jvm dtest:

    @Test
    public void testDroppingColumnWhenOneNodeIsDown() throws IOException, ExecutionException, InterruptedException
    {
        try (Cluster cluster = init(Cluster.build(3).start()))
        {
            cluster.schemaChange("create table " + KEYSPACE + ".tab (id int primary key, v1 int, v2 int)");
            cluster.disableAutoCompaction(KEYSPACE);

            cluster.coordinator(1).execute("insert into " + KEYSPACE + ".tab (id, v1, v2) VALUES (?, ?, ?)", ConsistencyLevel.ALL, 1, 2, 3);
            cluster.coordinator(1).execute("insert into " + KEYSPACE + ".tab (id, v1) VALUES (?, ?)", ConsistencyLevel.ALL, 4, 5);
            cluster.coordinator(1).execute("insert into " + KEYSPACE + ".tab (id, v2) VALUES (?, ?)", ConsistencyLevel.ALL, 6, 7);

            cluster.get(3).shutdown(true).get();

            cluster.schemaChangeIgnoringStoppedInstances("alter table " + KEYSPACE + ".tab drop v1");

            cluster.get(3).startup();

            cluster.coordinator(1).execute("insert into " + KEYSPACE + ".tab (id, v2) VALUES (?, ?)", ConsistencyLevel.ALL, 1, 9);
            cluster.coordinator(1).execute("insert into " + KEYSPACE + ".tab (id, v2) VALUES (?, ?)", ConsistencyLevel.ALL, 2, 11);

            NodeToolResult repairResult = cluster.get(1).nodetoolResult("repair", KEYSPACE, "tab");
            assertThat(repairResult.getRc()).isZero();

            SimpleQueryResult rows = cluster.coordinator(1).executeWithResult("select * from " + KEYSPACE + ".tab", ConsistencyLevel.ALL);
            assertRows(rows.toObjectArrays(), row(1, 9), row(2, 11), row(4, null), row(6, 7));

            assertRows(cluster.coordinator(1).executeWithResult("select * from " + KEYSPACE + ".tab where id = ?", ConsistencyLevel.ALL, 1).toObjectArrays(), row(1, 9));
            assertRows(cluster.coordinator(1).executeWithResult("select * from " + KEYSPACE + ".tab where id = ?", ConsistencyLevel.ALL, 2).toObjectArrays(), row(2, 11));
            assertRows(cluster.coordinator(1).executeWithResult("select * from " + KEYSPACE + ".tab where id = ?", ConsistencyLevel.ALL, 4).toObjectArrays(), row(4, null));
            assertRows(cluster.coordinator(1).executeWithResult("select * from " + KEYSPACE + ".tab where id = ?", ConsistencyLevel.ALL, 6).toObjectArrays(), row(6, 7));
        }
    }

Issue Links

• is related to: CASSANDRA-17047 Dropping a column can break queries until the schema is fully propagated (TAKE 2)