[CASSANDRA-19831] Improve JVMStabilityInspector error message to clarify off-heap vs heap OOM errors

• Type: Task
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Component/s: Local/Other
• Change Category: Operability
• Complexity: Low Hanging Fruit
• Platform: All
• Assignee: Unassigned
• Reporter: Brad Schoening
• Created: 12/Aug/24 21:05
• Updated: 26/Sep/24 20:41

Description

When we see Direct buffer memory errors, the JVMStabilityInspector appears to suggest it's a heap OOM to casual users.  This can confuse users who think that increasing the heap may help solve the problem. A vendors support staff suggested increasing the heap in this scenario.

A more clear error message could be something like:

Off-heap OOM forcing heap space OutOfMemoryError after OutOfMemoryError: Direct buffer memory

2024-08-01 17:00:00,16 [ERROR] [Messaging-EventLoop-3-6] cluster_id=001 ip_address=172.0.0.1  JVMStabilityInspector.java:151 - Force heap space OutOfMemoryError in the presence of
java.lang.OutOfMemoryError: Direct buffer memory
  at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
  at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
  at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
  at org.apache.cassandra.utils.MerkleTree.allocate(MerkleTree.java:742)
  at org.apache.cassandra.utils.MerkleTree.deserializeOffHeap(MerkleTree.java:780)

The comment in the code is clear, but 

  /**
     * Intentionally produce a heap space OOM upon seeing a non heap memory OOM.
     * Direct buffer OOM cannot trigger JVM OOM error related options,
     * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
     * See CASSANDRA-15214 and CASSANDRA-17128 for more details
     */

but the log message users see begins with 'Force heap space OutOfMemoryError', which suggests a heap memory issue, not off-heap memory.  Very confusing for general users and junior support staff.