[CASSANDRA-17298] Improve accuracy of memtable heap usage tracking

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.14, 4.1.7, 5.0.1, 5.1
• Component/s: Test/unit
• Bug Category: Correctness - Test Failure
• Severity: Low
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All
• Since Version: 4.0.0

Source Control Link

https://github.com/apache/cassandra/commit/c56ba3b317e67f4530db1737169f5558969bd531

Test and Documentation Plan

existing tests are adjusted

• Assignee: Dmitry Konstantinov
• Reporter: Josh McKenzie

Authors

Dmitry Konstantinov

Reviewers

Branimir Lambov, Stefan Miklosovic

• Created: 26/Jan/22 17:42
• Updated: 12/Sep/24 08:50
• Resolved: 10/Sep/24 09:23

Description

https://ci-cassandra.apache.org/job/Cassandra-4.0/313/testReport/org.apache.cassandra.cql3/MemtableSizeTest/testTruncationReleasesLogSpace_2/

 Failed 4 times in the last 30 runs. Flakiness: 27%, Stability: 86%
Error Message
Expected heap usage close to 49.930MiB, got 41.542MiB.

Stacktrace
junit.framework.AssertionFailedError: Expected heap usage close to 49.930MiB, got 41.542MiB.

	at org.apache.cassandra.cql3.MemtableSizeTest.testSize(MemtableSizeTest.java:130)
	at org.apache.cassandra.Util.runCatchingAssertionError(Util.java:644)
	at org.apache.cassandra.Util.flakyTest(Util.java:669)
	at org.apache.cassandra.cql3.MemtableSizeTest.testTruncationReleasesLogSpace(MemtableSizeTest.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 

UPDATE: It was discovered that unit tests were running with memtable_allocation_type: offheap_objects when we ship C* with heap_buffers.
So we changed that in CASSANDRA-19326, now we test with memtable_allocation_type: heap_buffers. As a result, this test now fails all the time on 4.0 and 4.1.

Issue Links

• relates to: CASSANDRA-16684 Flaky MemtableSizeTest
• relates to: CASSANDRA-19911 Removed MemtableSizeTest test re-appeared after a merge
• links to: GitHub Pull Request #3503
• links to: GitHub Pull Request #3504
• links to: GitHub Pull Request #3528
• links to: GitHub Pull Request #3529