[CASSANDRA-20058] Streamline the serialized format for index status gossip messages

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0.3, 5.1
• Component/s: Cluster/Gossip, Feature/2i Index
• Bug Category: Availability - Unavailable
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 5.0.0

Source Control Link

https://github.com/apache/cassandra/commit/376fe2a9fe3f13c7555c40cda6d3912d55ef63cc

Test and Documentation Plan

expanded on IndexAvailabilityTest to cover the mixed cluster scenarios we expect to see

• Assignee: Caleb Rackliffe
• Reporter: Jon Haddad

Authors

Caleb Rackliffe

Reviewers

David Capwell

• Created: 05/Nov/24 19:45
• Updated: 21/Nov/24 21:21
• Resolved: 21/Nov/24 21:20

Description

I spoke with someone today hitting an error on a cluster recently upgraded from 4.0.5 to 5.0.2, when creating or dropping tables.  Immediately after creating a table, nodes are reported down and they all immediately start throwing the below exception repeatedly in the logs.  The cluster has to be restarted to come back online.

Unfortunately I don't have access to the cluster, this was sent to me on a phone call.

Here's the exception:

 

ERROR [GossipStage:1] 2024-11-03 09:48:04,187 JVMStabilityInspector.java:70 - 
Exception in thread Thread[GossipStage:1,5,GossipStage]
java.lang.RuntimeException: java.lang.AssertionError
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
        at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
        at org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.AssertionError: null
        at org.apache.cassandra.db.TypeSizes.sizeof(TypeSizes.java:44)
        at org.apache.cassandra.gms.VersionedValue$VersionedValueSerializer.serializedSize(VersionedValue.java:365)
        at org.apache.cassandra.gms.VersionedValue$VersionedValueSerializer.serializedSize(VersionedValue.java:343)
        at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:344)
        at org.apache.cassandra.gms.EndpointStateSerializer.serializedSize(EndpointState.java:300)
        at org.apache.cassandra.gms.GossipDigestAckSerializer.serializedSize(GossipDigestAck.java:96)
        at org.apache.cassandra.gms.GossipDigestAckSerializer.serializedSize(GossipDigestAck.java:61)
        at org.apache.cassandra.net.Message$Serializer.payloadSize(Message.java:1088)
        at org.apache.cassandra.net.Message.payloadSize(Message.java:1131)
        at org.apache.cassandra.net.Message$Serializer.serializedSize(Message.java:769)
        at org.apache.cassandra.net.Message.serializedSize(Message.java:1111)
        at org.apache.cassandra.net.OutboundConnections.connectionTypeFor(OutboundConnections.java:215)
        at org.apache.cassandra.net.OutboundConnections.connectionFor(OutboundConnections.java:207)
        at org.apache.cassandra.net.OutboundConnections.enqueue(OutboundConnections.java:96)
        at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:473)
        at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70)
        at org.apache.cassandra.net.MessagingService.send(MessagingService.java:462)
        at org.apache.cassandra.net.MessagingService.send(MessagingService.java:437)
        at org.apache.cassandra.gms.GossipDigestSynVerbHandler.doVerb(GossipDigestSynVerbHandler.java:110)
        at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
        at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
        ... 7 common frames omitted

Issue Links

• links to: GitHub Pull Request #3678
• links to: GitHub Pull Request #3703