[CASSANDRA-19017] Ensure that empty SAI column indexes do not fail on validation after full-SSTable streaming

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-beta1, 5.0, 5.1
• Component/s: Consistency/Streaming, Feature/SAI
• Epic Link: Storage Attached Indexes (Phase 2)
• Bug Category: Correctness
• Severity: Critical
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 5.0-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/e8fb4b2f63b32f337447992f9eb57a12e2afc0e4

Test and Documentation Plan

PR	CI
5.0	j11 j17
trunk	j11 j17

• Assignee: Andres de la Peña
• Reporter: Caleb Rackliffe

Reviewers

Caleb Rackliffe

• Created: 09/Nov/23 20:23
• Updated: 05/Dec/23 12:51
• Resolved: 22/Nov/23 15:27

Description

I actually discovered this in some exploratory testing for CASSANDRA-19007...

@Test
public void testRepairWithEmptyColumnIndex() throws IOException
{
   try (Cluster cluster = init(Cluster.build(2).withConfig(config -> config.with(GOSSIP).with(NETWORK)).start()))
   {
       cluster.schemaChange(withKeyspace("CREATE TABLE %s.t (k int PRIMARY KEY, a int, b int)"));
       cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(a) USING 'sai'"));
       cluster.schemaChange(withKeyspace("CREATE INDEX ON %s.t(b) USING 'sai'"));

       // insert a split row
       cluster.get(1).executeInternal(withKeyspace("INSERT INTO %s.t(k, a) VALUES (0, 1)"));
       cluster.get(2).executeInternal(withKeyspace("INSERT INTO %s.t(k, b) VALUES (0, 2)"));
       cluster.get(1).flush(KEYSPACE);
       cluster.get(2).flush(KEYSPACE);
       cluster.get(1).nodetoolResult("repair", KEYSPACE).asserts().success(); // fails w/ no data for the "a" index on node 2!
   }
}

java.lang.RuntimeException: Repair job has failed with the error message: Repair command #1 failed with error Repair session 55a34e20-7f24-11ee-b345-d158c708a34e for range [(-1,9223372036854775805], (9223372036854775805,-1]] failed with error Stream failed: 
Session peer /127.0.0.2:7012 Failed because of an unknown exception
java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /private/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/dtests2185335212676803469/node1/data0/distributed_test_keyspace/t-3459c6aa21753083afcc5b0c9f77c3f7/distributed_test_keyspace-t-nc-4-big-SAI+aa+t_a_idx+Meta.db
	org.apache.cassandra.index.sai.disk.v1.V1OnDiskFormat.rethrowIOException(V1OnDiskFormat.java:241)
	org.apache.cassandra.index.sai.disk.v1.V1OnDiskFormat.validatePerColumnIndexComponents(V1OnDiskFormat.java:230)
java.nio.file.NoSuchFileException: /private/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/dtests2185335212676803469/node1/data0/distributed_test_keyspace/t-3459c6aa21753083afcc5b0c9f77c3f7/distributed_test_keyspace-t-nc-4-big-SAI+aa+t_a_idx+Meta.db
	java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)

This is an untested side-effect of a change in CASSANDRA-18670. The fix should be fairly simple, essentially making sure we don't assume the column index components are present during validation when the index completion marker is present.

Issue Links

• is caused by: CASSANDRA-18670 Importer should build SSTable indexes successfully before making new SSTables readable
• relates to: CASSANDRA-19007 Queries with multi-column replica-side filtering can miss rows
• links to: GitHub Pull Request #2904
• links to: GitHub Pull Request #2905