[CASSANDRA-17655] Prevent JVM Humongous allocations

• Type: Bug
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Platform: Java8, Java11
• Assignee: Unassigned
• Reporter: Brad Schoening
• Created: 23/May/22 17:20
• Updated: 09/Aug/22 19:41

Description

Several articles on Cassandra note that certain workloads, especially large partitions, can cause humongous allocations in Java, which can be reduced, but not always eliminated by increasing RegionSize.  RegionSize max is 32MB.  Humongous allocations are an undesirable performance issue because new objects get directly allocated in old space.

Below is a sample message seen in the gc.log

2022-05-23T18:46:58.743+0800: 3758624.170: [GC pause (G1 Humongous Allocation) (young) (initial-mark) 3758624.171: [G1Ergonomics (CSet Construction) start choosing CSet, _pending_cards: 56725, predicted base time: 39.83 ms, remaining time: 160.17 ms, target pause time: 200.00 ms]

Instead of Cassandra users tuning RegionSize manually, a better solution would be for Cassandra to limit the size of allocations, or consider moving humongous allocations off heap.

 

References

https://docs.oracle.com/en/java/javase/11/gctuning/garbage-first-g1-garbage-collector1.html#GUID-D74F3CC7-CC9F-45B5-B03D-510AEEAC2DAC

https://stevenlacerda.medium.com/identifying-and-fixing-humongous-allocations-in-cassandra-bf46444cec41

https://krzysztofslusarski.github.io/2020/11/10/humongous.html

Issue Links

• is related to: CASSANDRA-17408 Humongous Allocations after upgrading to 4.0