[CASSANDRA-17116] When streaming sees a ClosedChannelException this triggers the disk failure policy

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1-alpha1, 4.1
• Component/s: Consistency/Streaming
• Bug Category: Availability - Process Crash
• Severity: Normal
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/0859b3c4cfa690dd270030c4bc7ff93eadf6a732

Test and Documentation Plan

tests added

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell, Francisco Guerrero

Reviewers

Caleb Rackliffe, Dinesh Joshi

• Created: 02/Nov/21 23:38
• Updated: 08/Aug/23 16:42
• Resolved: 04/Feb/22 19:53

Description

Found in CASSANDRA-17085.

https://app.circleci.com/pipelines/github/dcapwell/cassandra/1069/workflows/26b7b83a-686f-4516-a56a-0709d428d4f2/jobs/7264
https://app.circleci.com/pipelines/github/dcapwell/cassandra/1069/workflows/26b7b83a-686f-4516-a56a-0709d428d4f2/jobs/7256

ERROR [Stream-Deserializer-/127.0.0.1:7000-f2eb1a15] 2021-11-02 21:35:40,983 DefaultFSErrorHandler.java:104 - Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.FSWriteError: java.nio.channels.ClosedChannelException
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:227)
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.writeComponent(BigTableZeroCopyWriter.java:206)
	at org.apache.cassandra.db.streaming.CassandraEntireSSTableStreamReader.read(CassandraEntireSSTableStreamReader.java:125)
	at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
	at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:51)
	at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:37)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
	at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:62)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException: null
	at org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:136)
	at org.apache.cassandra.net.AsyncStreamingInputPlus.consume(AsyncStreamingInputPlus.java:155)
	at org.apache.cassandra.io.sstable.format.big.BigTableZeroCopyWriter.write(BigTableZeroCopyWriter.java:217)
	... 9 common frames omitted

When bootstrap fails and streaming is closed, this triggers the disk failure policy which causes the JVM to halt by default (if this happens outside of bootstrap, then we stop transports and keep the JVM up).

org.apache.cassandra.streaming.StreamDeserializingTask attempts to handle this by ignoring this exception, but the call to org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize
Does try/catch and inspects exception; triggering this condition.

Issue Links

• is related to: CASSANDRA-17076 Fix failing test: test_bootstrap_with_reset_bootstrap_state - bootstrap_test.TestBootstrap
• is related to: CASSANDRA-17081 Fix test: bootstrap_test.py::TestBootstrap::test_bootstrap_with_reset_bootstrap_state
• is related to: CASSANDRA-17085 commit log was switched from non-daemon to daemon threads, which causes the JVM to exit in some case as no non-daemon threads are active
• is related to: CASSANDRA-18733 Waiting indefinitely on ReceivedMessage response in StreamSession#receive() can cause deadlock
• links to: Circle CI: trunk
• links to: GH PR: dtest trunk
• links to: GH PR: trunk