[CASSANDRA-19735] Cannot correctly create keyspace statement with replication during schemaChange

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Problem
• Fix Version/s: 4.1.x, 5.0-rc1, 5.0, 5.x
• Component/s: Cluster/Schema
• Platform: All
• Assignee: Unassigned
• Reporter: ConfX
• Created: 28/Jun/24 21:36
• Updated: 18/Jul/24 21:22
• Resolved: 02/Jul/24 07:09

Description

What happened

A specific schema change for creating keyspace with replications failed during Cassandra upgrade testing, but can pass under Cassandra distributed testing (non-upgrade).

How to reproduce:

Put the following test under cassandra/test/distributed/org/apache/cassandra/distributed/upgrade/, and build dtest jars for any versions within [4.1.3, 5.0-alpha2].

package org.apache.cassandra.distributed.upgrade;
public class demoUpgradeTest extends UpgradeTestBase
    @Test
    public void demoTest() throws Throwable {
        new TestCase()
                .nodes(1)
                .nodesToUpgrade(1)
                .withConfig(config -> config.with(NETWORK, GOSSIP, NATIVE_PROTOCOL))
                .upgradesToCurrentFrom(v41)
                .setup((cluster) -> {
                    cluster.schemaChange(withKeyspace("CREATE KEYSPACE %s WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2}"));
                }).runAfterNodeUpgrade((cluster, node) -> {
                    // let's do nothing here.
                }).run();
    }
} 

Run the test with

$ ant test-jvm-dtest-some-Duse.jdk11=true -Dtest.name=org.apache.cassandra.distributed.upgrade.demoUpgradeTest 

You will see the following failure:

[junit-timeout] Testcase: demoTest(org.apache.cassandra.distributed.upgrade.demoUpgradeTest)-_jdk11:    Caused an ERROR
[junit-timeout] Cannot add existing keyspace "distributed_test_keyspace"
[junit-timeout] org.apache.cassandra.exceptions.AlreadyExistsException: Cannot add existing keyspace "distributed_test_keyspace"
[junit-timeout]     at org.apache.cassandra.cql3.statements.schema.CreateKeyspaceStatement.apply(CreateKeyspaceStatement.java:78)
[junit-timeout]     at org.apache.cassandra.schema.DefaultSchemaUpdateHandler.apply(DefaultSchemaUpdateHandler.java:230)
[junit-timeout]     at org.apache.cassandra.schema.Schema.transform(Schema.java:597)
[junit-timeout]     at org.apache.cassandra.cql3.statements.schema.AlterSchemaStatement.execute(AlterSchemaStatement.java:114)
[junit-timeout]     at org.apache.cassandra.cql3.statements.schema.AlterSchemaStatement.execute(AlterSchemaStatement.java:60)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:122)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:103)
[junit-timeout]     at org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:66)
[junit-timeout]     at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
[junit-timeout]     at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
[junit-timeout]     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit-timeout]     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit-timeout]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[junit-timeout]     at java.base/java.lang.Thread.run(Thread.java:829) 

I have tested version pairs 4.1.3_4.1.4, 4.1.4_4.1.5, 4.1.5_5.0-alpha1, and 5.0-alpha1_5.0-alpha2. All of them have the same issue.

I wrote a very similar test with Cassandra distributed test framework (non-upgrade test) as below:

package org.apache.cassandra.distributed.test.streaming;public class LCSStreamingKeepLevelTest extends TestBaseImpl
{
    @Test
    public void demoTest() throws IOException
    {
        try (Cluster cluster = builder().withNodes(1)
                .withConfig(config -> config.with(NETWORK, GOSSIP, NATIVE_PROTOCOL))
                .start())
        {
            cluster.schemaChange(withKeyspace("CREATE KEYSPACE %s WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2}"));
        }
    }
} 

This distributed test would pass successfully without any issues.

 

The expected behavior should be that the upgrade test above can also perform cluster.schemaChange() successfully.