[CASSANDRA-11286] streaming socket never times out

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.14, 2.2.6, 3.0.5, 3.5
• Component/s: Legacy/Streaming and Messaging
• Severity: Normal
• Assignee: Paulo Motta
• Reporter: Paulo Motta

Authors

Paulo Motta

Reviewers

Yuki Morishita

• Created: 01/Mar/16 03:22
• Updated: 16/Apr/19 09:30
• Resolved: 10/Mar/16 19:05

Description

While trying to reproduce CASSANDRA-8343 I was not able to trigger a SocketTimeoutException by adding an artificial sleep longer than streaming_socket_timeout_in_ms.

After investigation, I detected two problems:

• ReadableByteChannel creation via socket.getChannel(), as done in ConnectionHandler.getReadChannel(socket), does not respect socket.setSoTimeout(), as explained in this blog post

  • The only difference between “blocking NIO” and “NIO wrapped around IO” is that you can’t use socket timeout with SocketChannels. Why ? Read a javadoc for setSocketTimeout(). It says that this timeout is used only by streams.

• socketSoTimeout is never set on "follower" side, only on initiator side via DefaultConnectionFactory.createConnection(peer).

This may cause streaming to hang indefinitely, as exemplified by CASSANDRA-8621:

For the scenario that prompted this ticket, it appeared that the streaming process was completely stalled. One side of the stream (the sender side) had an exception that appeared to be a connection reset. The receiving side appeared to think that the connection was still active, at least in terms of the netstats reported by nodetool. We were unable to verify whether this was specifically the case in terms of connected sockets due to the fact that there were multiple streams for those peers, and there is no simple way to correlate a specific stream to a tcp session.