[CASSANDRA-15388] Add compaction allocation measurement test to support compaction gc optimization.

• Type: Sub-task
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-alpha4, 4.0
• Component/s: Local/Compaction
• Change Category: Performance
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra/commit/c9e40277f0389a773b21d4310a85a8c761932d61

Test and Documentation Plan

circle

• Assignee: Blake Eggleston
• Reporter: Blake Eggleston

Authors

Blake Eggleston

Reviewers

Benedict Elliott Smith, David Capwell

• Created: 30/Oct/19 22:30
• Updated: 03/Oct/22 22:13
• Resolved: 11/Mar/20 19:53

Description

This adds a test that is able to quickly and accurately measure the effect of potential gc optimizations against a wide range of (synthetic) compaction workloads. This test accurately measures allocation rates from 16 workloads in less that 2 minutes.

This test uses google’s java-allocation-instrumenter agent to measure the workloads. Measurements using this agent are very accurate and pretty repeatable from run to run, with most variance being negligible (1-2 bytes per partition), although workloads with larger but fewer partitions vary a bit more (still less that 0.03%).

The thinking behind this patch is that with compaction, we’re generally interested in the memory allocated per partition, since garbage scales more or less linearly with the number of partitions compacted. So measuring allocation from a small number of partitions that otherwise represent real world use cases is a good enough approximation.

In addition to helping with compaction optimizations, this test could be used as a template for future optimization work. This pattern could also be used to set allocation limits on workloads/operations and fail CI if the allocation behavior changes past some threshold.

Issue Links

• causes: CASSANDRA-16234 Update NetBeans project file for dependency changes since 11th Feb 2020
• relates to: CASSANDRA-17947 Remove test-memory target