[CASSANDRA-17810] Revise timeout handling on queries triggering timeout introduced by CASSANDRA-7392

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-alpha1, 5.0
• Component/s: Messaging/Internode
• Bug Category: Degradation - Other Exception
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.2

Source Control Link

https://gitbox.apache.org/repos/asf?p=cassandra.git;a=commit;h=f4b69ba0e82bb051e56a92d792142034d9f617f0

Test and Documentation Plan

New in-jvm distributed testing of the newly introduced exception + modification of unit tests to confirm it working

• Assignee: Josh McKenzie
• Reporter: Josh McKenzie

Authors

Josh McKenzie

Reviewers

Aleksey Yeschenko

• Created: 11/Aug/22 16:13
• Updated: 12/Sep/23 13:03
• Resolved: 26/Sep/22 17:58

Description

In situations where a replica times out with the code introduced in CASSANDRA-7392, they drop the timed-out query on the floor without notifying the coordinator; this doesn't become apparent coordinator-side until we go to serialize the response. While the results of this (i.e. missed data) should mostly be fixed by short read protection on repeat requests, the coordinator can in some cases throw an exception like IllegalStateException: UnfilteredRowIterator for keyspace.table has an open RT bound as its last item which is decidedly unfriendly.

We should introduce a specific exception to allow replicas to notify a coordinator something has timed out and been dropped on the floor so the coordinator can appropriately handle that scenario and abort the query at that time.