[CASSANDRA-15452] Improve disk access patterns during compaction (big format)

• Type: Improvement
• Status: Review In Progress
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 4.1.x, 5.0.x, 5.x
• Component/s: Legacy/Local Write-Read Paths, Local/Compaction
• Change Category: Performance
• Complexity: Normal
• Platform: All

Test and Documentation Plan

TBD

• Assignee: Jordan West
• Reporter: Jon Haddad

Authors

Jordan West

Reviewers

Caleb Rackliffe

• Created: 13/Dec/19 16:53
• Updated: 02/Dec/24 10:00

Description

On read heavy workloads Cassandra performs much better when using a low read ahead setting.   In my tests I've seen an 5x improvement in throughput and more than a 50% reduction in latency.  However, I've also observed that it can have a negative impact on compaction and streaming throughput. It especially negatively impacts cloud environments where small reads incur high costs in IOPS due to tiny requests.

1. We should investigate using POSIX_FADV_DONTNEED on files we're compacting to see if we can improve performance and reduce page faults. 
2. This should be combined with an internal read ahead style buffer that Cassandra manages, similar to a BufferedInputStream but with our own machinery.  This buffer should read fairly large blocks of data off disk at at time.  EBS, for example, allows 1 IOP to be up to 256KB.  A considerable amount of time is spent in blocking I/O during compaction and streaming. Reducing the frequency we read from disk should speed up all sequential I/O operations.
3. We can reduce system calls by buffering writes as well, but I think it will have less of an impact than the reads

Issue Links

• is duplicated by: CASSANDRA-19607 Compaction double reads every chunk
• is duplicated by: CASSANDRA-19494 Optimize I/O during table scans
• links to: GitHub Pull Request #3606