[CASSANDRA-14507] OutboundMessagingConnection backlog is not fully written in case of race conditions

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Feedback Received
• Component/s: Legacy/Streaming and Messaging
• Severity: Normal
• Assignee: Unassigned
• Reporter: Sergio Bossa
• Created: 08/Jun/18 11:49
• Updated: 16/Apr/19 09:29
• Resolved: 22/Aug/18 11:46

Description

The OutboundMessagingConnection writes into a backlog queue before the connection handshake is successfully completed, and then writes such backlog to the channel as soon as the successful handshake moves the channel state to READY.
This is unfortunately race prone, as the following could happen:
1) One or more writer threads see the channel state as NOT_READY in #sendMessage() and are about to enqueue to the backlog, but they get descheduled by the OS.
2) The handshake thread is scheduled by the OS and moves the channel state to READY, emptying the backlog.
3) The writer threads are scheduled back and add to the backlog, but the channel state is READY at this point, so those writes would sit in the backlog and expire.

Please note a similar race condition exists between OutboundMessagingConnection#sendMessage() and MessageOutHandler#channelWritabilityChanged(), which is way more serious as the channel writability could frequently change, luckily it looks like ChannelWriter#write() never gets invoked with checkWritability at true (so writes never go to the backlog when the channel is not writable).

Issue Links

• duplicates: CASSANDRA-14503 Internode connection management is race-prone