[CASSANDRA-16157] RTE during re-serialization for message filtering during 3.0 -> 4.0 upgrade

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta3, 4.0
• Component/s: Test/dtest/java
• Bug Category: Availability - Unavailable
• Severity: Critical
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0-beta1

Source Control Link

https://github.com/apache/cassandra/commit/5be83b6a72695253c552535d2b826209f144cc63

Test and Documentation Plan

Test included

• Assignee: Alex Petrov
• Reporter: Alex Petrov

Authors

Alex Petrov

Reviewers

David Capwell, Yifan Cai

• Created: 01/Oct/20 14:58
• Updated: 03/Jan/21 17:03
• Resolved: 14/Oct/20 08:59

Description

When trying to upgrade 3.0 to 4.0, we’re often running into a problem, if older node serves as a coordinator:

 15294 java.lang.RuntimeException: Can not deserialize message org.apache.cassandra.distributed.impl.MessageImpl@4c46aead
  15295         at org.apache.cassandra.distributed.impl.Instance.deserializeMessage(Instance.java:299) ~[dtest-4.0-beta3.jar:?]
  15296         at org.apache.cassandra.distributed.impl.Instance.lambda$receiveMessage$7(Instance.java:315) ~[dtest-4.0-beta3.jar:?]
  15297         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_232]
  15298         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_232]
  15299         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_232]
  15300         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_232]
  15301         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [dtest-4.0-beta3.jar:?]
  15302         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
  15303 Caused by: java.io.EOFException
  15304         at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:180) ~[dtest-4.0-beta3.jar:?]
  15305         at org.apache.cassandra.utils.vint.VIntCoding.readUnsignedVInt(VIntCoding.java:68) ~[dtest-4.0-beta3.jar:?]
  15306         at org.apache.cassandra.io.util.RebufferingInputStream.readUnsignedVInt(RebufferingInputStream.java:243) ~[dtest-4.0-beta3.jar:?]
  15307         at org.apache.cassandra.net.Message$Serializer.deserializeHeaderPost40(Message.java:694) ~[dtest-4.0-beta3.jar:?]
  15308         at org.apache.cassandra.net.Message$Serializer.deserializePost40(Message.java:765) ~[dtest-4.0-beta3.jar:?]
  15309         at org.apache.cassandra.net.Message$Serializer.deserialize(Message.java:625) ~[dtest-4.0-beta3.jar:?]
  15310         at org.apache.cassandra.distributed.impl.Instance.deserializeMessage(Instance.java:295) ~[dtest-4.0-beta3.jar:?]
  15311         ... 7 more

Issue Links

• blocks: CASSANDRA-16207 NPE when calling broadcast address on unintialized node
• blocks: CASSANDRA-15935 Improve machinery for testing consistency in presence of range movements