[CASSANDRA-18471] CEP-15 Accord: NotWitnessed commands can receive an invalidate promise but would return Zero instead

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-alpha1, 5.0
• Component/s: Accord
• Labels: pull-request-available
• Bug Category: Availability - Response Crash
• Severity: Normal
• Complexity: Low Hanging Fruit
• Discovered By: Fuzz Test
• Platform: All
• Since Version: NA

Source Control Link

https://github.com/apache/cassandra-accord/commit/33c91e4489597c1e88df8a5254c015eeb76899f1

Test and Documentation Plan

new test

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell

Reviewers

Benedict Elliott Smith

• Created: 20/Apr/23 23:04
• Updated: 27/Oct/23 15:46
• Resolved: 25/Apr/23 16:39

Description

While working on CASSANDRA-18451 I hit the following failure

Failed on seed -5929214838499924343
accord.burn.SimulationException: Failed on seed -5929214838499924343
Caused by: java.lang.AssertionError: Unexpected exception encountered
	at accord.impl.basic.PropagatingPendingQueue.poll(PropagatingPendingQueue.java:73)
	at accord.impl.basic.Cluster.processPending(Cluster.java:179)
	at accord.impl.basic.Cluster.run(Cluster.java:296)
	at accord.burn.BurnTest.burn(BurnTest.java:309)
	at accord.burn.BurnTest.run(BurnTest.java:386)
	at accord.burn.BurnTest.testOne(BurnTest.java:372)
	Suppressed: java.lang.IllegalStateException: Received replies from a node that must have known the route, but that did not include it
		at accord.coordinate.Invalidate.invalidate(Invalidate.java:204)
		at accord.coordinate.Invalidate.handle(Invalidate.java:131)
		at accord.coordinate.Invalidate.onSuccess(Invalidate.java:105)
		at accord.coordinate.Invalidate.onSuccess(Invalidate.java:51)
		at accord.impl.basic.Cluster.lambda$processNext$1(Cluster.java:209)
		at accord.impl.basic.Cluster.now(Cluster.java:260)
		at accord.impl.basic.Cluster.processNext(Cluster.java:206)
		at accord.impl.basic.Cluster.processPending(Cluster.java:183)

In a debugger was able to figure out the state and create a unit test to hit the same situation

class InvalidateTest
{
    @Test
    void test() throws ExecutionException
    {
        try (MockCluster cluster = MockCluster.builder().replication(2).nodes(2).build())
        {
            Node n1 = cluster.get(1);
            Node n2 = cluster.get(2);

            RoutingKey n1RoutingKey = n1.topology().current().get(0).range.end();
            IntKey.Raw n1key = IntKey.key(((IntKey.Routing) n1RoutingKey).key);

            RoutingKey n2RoutingKey = n1.topology().current().get(1).range.end();
            IntKey.Raw n2key = IntKey.key(((IntKey.Routing) n2RoutingKey).key);

            Keys keys = Keys.of(n1key, n2key);


            Node coordinator = n1;
            TxnId txnId = coordinator.nextTxnId(Txn.Kind.Read, Routable.Domain.Key);
            Txn txn = readOnly(keys);

            AsyncChains.getUninterruptibly(n2.commandStores().unsafeForKey(n2key).execute(PreLoadContext.contextFor(txnId, keys), store -> {
                Ranges ranges = store.ranges().currentRanges();
                PartialTxn partial = txn.slice(ranges, true);
                FullKeyRoute route = keys.toRoute(n2RoutingKey);
//                RoutingKey progressKey = n2RoutingKey.toUnseekable(); // if this is non-null this passes
                RoutingKey progressKey = null;
                CheckedCommands.preaccept(store, txnId, partial, route, progressKey);
                CheckedCommands.accept(store, txnId, Ballot.ZERO, route.slice(ranges), partial.keys().slice(ranges), progressKey, txnId, PartialDeps.builder(ranges).build());
            }));
            AsyncChains.getUninterruptibly(new AsyncChains.Head<Outcome>() {
                @Override
                protected void start(BiConsumer<? super Outcome, Throwable> callback) {
                    Invalidate.invalidate(coordinator, txnId, keys.toUnseekables(), callback);
                }
            });
        }
    }

    private static Txn readOnly(Seekables<?, ?> keys)
    {
        Read read = MockStore.read(keys);
        Query query = Mockito.mock(Query.class);
        return new Txn.InMemory(keys, read, query);
    }
}

Issue Links

• links to: CI: accord trunk
• links to: GH: accord trunk
• links to: GitHub Pull Request #44