[CASSANDRA-19477] Do not go to disk to get HintsStore.getTotalFileSize

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.1.5, 5.0-beta2, 5.0, 5.1-alpha1
• Component/s: Consistency/Hints
• Bug Category: Code
• Severity: Normal
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 4.1-alpha1

Source Control Link

https://github.com/apache/cassandra/commit/38408938ccfe5b8c051e25c645bdcd71b45fa66e

Test and Documentation Plan

ci

• Assignee: Stefan Miklosovic
• Reporter: Jon Haddad

Authors

Stefan Miklosovic

Reviewers

Aleksey Yeschenko

• Created: 16/Mar/24 20:30
• Updated: 26/Mar/24 10:38
• Resolved: 25/Mar/24 13:35

Description

When testing a cluster with more requests than it could handle, I noticed significant CPU time (25%) spent in HintsStore.getTotalFileSize.  Here's what I'm seeing from profiling:

10% of CPU time spent in HintsDescriptor.fileName which only does this:

 

return String.format("%s-%s-%s.hints", hostId, timestamp, version);

At a bare minimum here we should create this string up front with the host and version and eliminate 2 of the 3 substitutions, but I think it's probably faster to use a StringBuilder and avoid the underlying regular expression altogether.

12% of the time is spent in org.apache.cassandra.io.util.File.length.  It looks like this is called once for each hint file on disk for each host we're hinting to.  In the case of an overloaded cluster, this is significant.  It would be better if we were to track the file size in memory for each hint file and reference that rather than go to the filesystem.

These fairly small changes should make Cassandra more reliable when under load spikes.

CPU Flame graph attached.

I only tested this in 4.1 but it looks like this is present up to trunk.

 

Issue Links

• fixes: CASSANDRA-19485 if max_hints_size_per_host < max_hints_file_size then it will write hints after max_hints_size_per_host is reached
• links to: GitHub Pull Request #3180
• links to: GitHub Pull Request #3192
• links to: GitHub Pull Request #3193