[CASSANDRA-7886] Coordinator should not wait for read timeouts when replicas hit Exceptions

• Type: Improvement
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 2.2.0 beta 1
• Labels: protocolv4 qa-resolved

Environment

Tested with Cassandra 2.0.8

• Impacts: Clients
• Assignee: Christian Spriegel
• Reporter: Christian Spriegel

Authors

Christian Spriegel

Reviewers

Tom Hobbs

• Tester: Alan Boudreault
• Created: 05/Sep/14 09:42
• Updated: 06/Nov/19 21:57
• Resolved: 09/Jan/15 19:34

Description

Issue
When you have TombstoneOverwhelmingExceptions occuring in queries, this will cause the query to be simply dropped on every data-node, but no response is sent back to the coordinator. Instead the coordinator waits for the specified read_request_timeout_in_ms.

On the application side this can cause memory issues, since the application is waiting for the timeout interval for every request.Therefore, if our application runs into TombstoneOverwhelmingExceptions, then (sooner or later) our entire application cluster goes down

Proposed solution
I think the data nodes should send a error message to the coordinator when they run into a TombstoneOverwhelmingException. Then the coordinator does not have to wait for the timeout-interval.

Issue Links

• is related to: CASSANDRA-8592 Add WriteFailureException
• relates to: CASSANDRA-6117 Avoid death-by-tombstone by default
• relates to: CASSANDRA-8561 Tombstone log warning does not log partition key
• relates to: CASSANDRA-6747 MessagingService should handle failures on remote nodes.
• requires: CASSANDRA-8043 Native Protocol V4