[CASSANDRA-15214] Internode messaging catches OOMs and does not rethrow

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0, 4.0-beta4
• Component/s: Messaging/Client, Messaging/Internode
• Bug Category: Availability - Process Crash
• Severity: Normal
• Complexity: Low Hanging Fruit
• Discovered By: Code Inspection
• Platform: All
• Since Version: 4.0-beta2

Source Control Link

https://github.com/apache/cassandra/commit/46ee939b957528185dc6bbd3028c1d6e695163e7

Test and Documentation Plan

ci

• Assignee: Yifan Cai Assign to me
• Reporter: Benedict Elliott Smith

Authors

Yifan Cai

Reviewers

David Capwell, Jordan West

• Created: 16/Jul/19 09:31
• Updated: 03/Jan/21 16:58
• Resolved: 19/Nov/20 19:22

Description

Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, so presently there is no way to ensure that an OOM reaches the JVM handler to trigger a crash/heapdump.

It may be that the simplest most consistent way to do this would be to have a single thread spawned at startup that waits for any exceptions we must propagate to the Runtime.

We could probably submit a patch upstream to Netty, but for a guaranteed future proof approach, it may be worth paying the cost of a single thread.