[CASSANDRA-15963] Flaky dtest test_dead_sync_initiator - repair_tests.repair_test.TestRepair

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta3, 4.0
• Component/s: Test/dtest/python
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

https://github.com/apache/cassandra-dtest/commit/3f34166bbeeac179acdf4874785435ae7442fd77

Test and Documentation Plan

none

• Assignee: Adam Holmberg
• Reporter: Yifan Cai

Authors

Adam Holmberg

Reviewers

Brandon Williams

• Created: 20/Jul/20 21:20
• Updated: 09/Apr/22 13:45
• Resolved: 21/Sep/20 20:38

Description

CI test link. https://app.circleci.com/pipelines/github/yifan-c/cassandra/78/workflows/e89b96f1-c468-484f-9764-694b64fad95b/jobs/411

test teardown failure
 Unexpected error found in node logs (see stdout for full details). Errors: [ERROR [Stream-Deserializer-127.0.0.1:7000-56da8052] 2020-07-20 19:00:33,545 CassandraEntireSSTableStreamReader.java:145 - [Stream 48a43e80-cabb-11ea-a743-3578c769ed44] Error while reading sstable from stream for table = keyspace1.standard1
 org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /tmp/dtest-4ichr2ha/test/node3/data0/keyspace1/standard1-3b1f65f0cabb11eaa7433578c769ed44/na-1-big-Statistics.db
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.maybeValidateChecksum(MetadataSerializer.java:219)
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:198)
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:129)
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.mutate(MetadataSerializer.java:226)
 at org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:140)
 at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
 at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:53)
 at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:38)
 at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
 at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:172)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.lang.Thread.run(Thread.java:748)
 Caused by: java.io.IOException: Checksums do not match for /tmp/dtest-4ichr2ha/test/node3/data0/keyspace1/standard1-3b1f65f0cabb11eaa7433578c769ed44/na-1-big-Statistics.db
 ... 12 common frames omitted, ERROR [Stream-Deserializer-127.0.0.1:7000-56da8052] 2020-07-20 19:00:33,545 CassandraEntireSSTableStreamReader.java:145 - [Stream 48a43e80-cabb-11ea-a743-3578c769ed44] Error while reading sstable from stream for table = keyspace1.standard1
 org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /tmp/dtest-4ichr2ha/test/node3/data0/keyspace1/standard1-3b1f65f0cabb11eaa7433578c769ed44/na-1-big-Statistics.db
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.maybeValidateChecksum(MetadataSerializer.java:219)
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:198)
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:129)
 at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.mutate(MetadataSerializer.java:226)
 at org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:140)
 at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
 at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:53)
 at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:38)
 at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
 at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:172)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.lang.Thread.run(Thread.java:748)
 Caused by: java.io.IOException: Checksums do not match for /tmp/dtest-4ichr2ha/test/node3/data0/keyspace1/standard1-3b1f65f0cabb11eaa7433578c769ed44/na-1-big-Statistics.db
... 12 common frames omitted]

Issue Links

• duplicates: CASSANDRA-15861 Mutating sstable component may race with entire-sstable-streaming(ZCS) causing checksum validation failure
• is duplicated by: CASSANDRA-14151 [TRUNK] TestRepair.test_dead_sync_initiator failed due to ERROR in logs "SSTableTidier ran with no existing data file for an sstable that was not new"
• is related to: CASSANDRA-17541 Test Failure: repair_tests.repair_test.TestRepair.test_dead_sync_initiator