[CASSANDRA-19383] Restarting a node causes MV unavailable exceptions afterward

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 4.0.x, 4.1.x, 5.0.x, 5.x
• Component/s: Feature/Materialized Views
• Bug Category: Availability - Unavailable
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Unassigned
• Reporter: Brandon Williams
• Created: 09/Feb/24 12:21
• Updated: 17/May/24 19:19

Description

If a cluster is started with materialized views enabled and a client is constantly writing to one, when a node is restarted shortly after startup it may throw UEs:

INFO [GossipStage:1] 2024-02-09 06:17:10,073 Gossiper.java:1419 - Node /10.5.11.102:7000 has restarted, now UP
ERROR [MutationStage-2] 2024-02-09 06:17:10,074 Keyspace.java:650 - Unknown exception caught while attempting to update MaterializedView! stresstest.transport_orders
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicas(ReplicaPlans.java:169)
at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicasForWrite(ReplicaPlans.java:112)
at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:353)
at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:344)
at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:338)
at org.apache.cassandra.service.StorageProxy.wrapViewBatchResponseHandler(StorageProxy.java:1417)
at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:1077)
at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:169)
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:645)
at org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:476)
at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:223)
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)

It's not immediately clear what the impact of this is since the error is on restart and not communicated back to the client.