[CASSANDRA-19900] Make the compression cache configurable to reduce heap pressure for large SSTables

• Type: Improvement
• Status: Resolved
• Priority: High
• Resolution: Fixed
• Fix Version/s: NA
• Component/s: Analytics Library
• Change Category: Performance
• Complexity: Low Hanging Fruit
• Platform: All

Source Control Link

https://github.com/apache/cassandra-analytics/commit/bff083e35ebc338daf93c1a3553c590ae1864115

Test and Documentation Plan

ci

• Assignee: James Berragan
• Reporter: James Berragan

Authors

James Berragan

Reviewers

Francisco Guerrero, Yifan Cai

• Created: 05/Sep/24 20:37
• Updated: 09/Sep/24 07:34
• Resolved: 08/Sep/24 15:14

Description

Normally the CompressionInfo.db file is trivial in size (a few kb) and so can be easily cached to prevent wasteful requests for the same file, but for large SSTables (running into a few hundred GB, due to schema issues or large blobs) then the CompressionInfo.db files can also grow to be a few hundred MB or GB making it problematic to cache. We should make the compression cache optional and configurable so we don’t cache large files.

Issue Links

• links to: CircleCI
• links to: GitHub PR
• links to: GitHub Pull Request #77