[CASSANDRA-13608] Connection closed/reopened during join with MVs causes Cassandra stream to close

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Legacy/Streaming and Messaging

Environment

Cassandra 3.10. Windows Server 2016, 32GB ram, 2TB hard disk, RAID10 with 4 spindles, 8 Cores

• Severity: Normal
• Since Version: 3.10
• Assignee: Kurt Greaves
• Reporter: Tania S Engel

Authors

Kurt Greaves

• Created: 15/Jun/17 15:23
• Updated: 16/Apr/19 09:30

Description

We use MVs. We start a JOIN bootstrap. Primary seed node streams to the replica. The replica requires some GC cleanup and experiences frequent pauses including a 12 second old gen cleanup following a memTable flush. Both replica and primary show MessagingService IOException: An existing connection was forcibly closed by the remote host. The replica MessagingService-Outgoing reestablishes the connection immediately but the primary StreamKeepAliveExecutor throws a java.RuntimeException: Outgoing stream handler has been closed. From that point forward, the replica stays in JOIN mode, sending keeping alive to the primary. The primary receives the keep alive, but does not send its own and it repeatedly fails to send a hints file to the replica. It seems this limping condition would continue indefinitely, but stops as we stop the replica Cassandra. If we restart the replica Cassandra the JOIN picks up again but fails with java.io.IOException: Corrupt value length 355151036 encountered, as it exceeds the maximum of 268435456, which is set via max_value_size_in_mb in cassandra.yaml. We have not increased this value as we do not have values that large in our data so we presume it is indeed corrupt and moving past it would not be a good idea. Please see the attachment for details.

ERROR [BatchlogTasks:1] 2017-06-13 20:24:13,953 CassandraDaemon.java:229 - Exception in thread Thread[BatchlogTasks:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: C:\Cassandra\data\data\system\batches-919a4bc57a333573b03e13fc3f68b465\mc-2-big-Data.db
	at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:395) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:257) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:69) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:67) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:26) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:96) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:755) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:446) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.cql3.UntypedResultSet$FromPager$1.computeNext(UntypedResultSet.java:193) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.cql3.UntypedResultSet$FromPager$1.computeNext(UntypedResultSet.java:179) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.batchlog.BatchlogManager.processBatchlogEntries(BatchlogManager.java:233) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.batchlog.BatchlogManager.replayFailedBatches(BatchlogManager.java:209) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-3.10.jar:3.10]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_66]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_66]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_66]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.10.jar:3.10]
	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_66]
Caused by: java.io.IOException: Corrupt value length 355151036 encountered, as it exceeds the maximum of 268435456, which is set via max_value_size_in_mb in cassandra.yaml
	at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:428) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:636) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:577) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1222) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1177) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.Columns.apply(Columns.java:377) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:571) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.readNext(UnfilteredDeserializer.java:211) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:144) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:168) ~[apache-cassandra-3.10.jar:3.10]
	at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:382) ~[apache-cassandra-3.10.jar:3.10]

Issue Links

• is related to: CASSANDRA-12783 Break up large MV mutations to prevent OOMs