[CASSANDRA-15191] stop_paranoid disk failure policy is ignored on CorruptSSTableException after node is up

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.22, 3.11.8, 4.0-beta2, 4.0
• Component/s: Local/Config
• Bug Category: Correctness
• Severity: Normal
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/db034609554a3185c0808cc67e9f0c148cc912c4

Test and Documentation Plan

jvm dtest

• Assignee: Stefan Miklosovic
• Reporter: Vincent White

Authors

Stefan Miklosovic

Reviewers

Brandon Williams, David Capwell

• Created: 01/Jul/19 04:45
• Updated: 09/Mar/23 16:09
• Resolved: 30/Jul/20 00:31

Description

There is a bug when disk_failure_policy is set to stop_paranoid and CorruptSSTableException is thrown after server is up. The problem is that this setting is ignored. Normally, it should stop gossip and transport but it just continues to serve requests and an exception is just logged.

 

This patch unifies the exception handling in JVMStabilityInspector and code is reworked in such way that this inspector acts as a central place where such exceptions are inspected. 

 

The core reason for ignoring that exception is that thrown exception in AbstractLocalAwareExecturorService is not CorruptSSTableException but it is RuntimeException and that exception is as its cause. Hence it is better if we handle this in JVMStabilityInspector which can recursively examine it, hence act accordingly.

Behaviour before:

stop_paranoid of disk_failure_policy is ignored when CorruptSSTableException is thrown, e.g. on a regular select statement

Behaviour after:

Gossip and transport (cql) is turned off, JVM is still up for further investigation e.g. by jmx.

Issue Links

• causes: CASSANDRA-18294 die disk failure policy will not kill jvm as documented