[CASSANDRA-16471] DataOutputBuffer#scratchBuffer can use off-heap or on-heap memory as a means to control memory allocations

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-alpha1, 5.0
• Component/s: Local/Caching
• Change Category: Performance
• Complexity: Low Hanging Fruit
• Platform: All

Source Control Link

https://github.com/apache/cassandra/commit/c1f28b85673298806c57e1eeeaf2d81a8ed4f1b8

Test and Documentation Plan

unit/perf tests

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell

Reviewers

Caleb Rackliffe

• Created: 25/Feb/21 23:45
• Updated: 12/Sep/23 13:03
• Resolved: 13/Aug/22 03:04

Description

While running workflows to compare 3.0 with trunk we found that allocations and GC are significantly higher for a write mostly workload (22% read, 3% delete, 75% write); below is what we saw for a 2h run

Allocations
30: 1.64TB
40: 2.99TB

GC Events
30: 7.39k events
40: 13.93k events

When looking at the allocation output we saw the follow for memory allocations

Here we see that org.apache.cassandra.io.util.DataOutputBuffer#expandToFit is around 52% of the memory allocations. When looking at this logic I see that allocations are on-heap and constantly throw away the buffer (as a means to allow GC to clean up).

With the patch, allocations/gc are the following

Allocations
30: 1.64TB
40 w/ patch: 1.77TB
40: 2.99TB

GC Events
30: 7.39k events
40 w/ patch: 8k events
40: 13.93k events

With the patch only 0.8% allocations

Issue Links

• links to: CI: trunk attempt 2
• links to: Circle CI: trunk
• links to: GH Branch: trunk
• links to: GH: trunk attempt 2