[CASSANDRA-17223] Node can be left in a bad state after invalid memory access exception

• Type: Bug
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Component/s: Local/Other
• Labels: lhf
• Platform: All
• Assignee: Unassigned
• Reporter: Paulo Motta
• Created: 21/Dec/21 02:15
• Updated: 05/Jan/22 15:39

Description

There were a few reports of errors containing the following exception:

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java 

(CASSANDRA-8642, CASSANDRA-5737, CASSANDRA-11201, CASSANDRA-13091), normally closed as invalid due to the difficulty in reproducing it.

This exception can be thrown in a variety of scenarios generally involving a kernel fault during unsafe memory access.

I've seen an occurence of this issue where the node went into a bad state after a memory access error.

I believe this exception should be treated analogous to `OutOfMemoryException` (CASSANDRA-13006 ) and cause the process to crash if/when encountered, perhaps guarded by a flag.

Issue Links

• is related to: CASSANDRA-5737 CassandraDaemon - recent unsafe memory access operation in compiled Java code
• is related to: CASSANDRA-8642 Cassandra crashed after stress test of write
• is related to: CASSANDRA-11201 Compaction memory fault in 3.0.3
• is related to: CASSANDRA-13091 SASIIndexTest.testTableRebuild throws unsafe memory access exception