[CASSANDRA-19369] [Analytics] Use XXHash32 for digest calculation of SSTables

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: NA
• Component/s: Analytics Library
• Change Category: Performance
• Complexity: Low Hanging Fruit
• Platform: All

Source Control Link

https://github.com/apache/cassandra-analytics/commit/dc0e79b9c483562ec0920d69e886715eb329c426

Test and Documentation Plan

Added unit tests and integration tests

• Assignee: Francisco Guerrero
• Reporter: Francisco Guerrero

Authors

Francisco Guerrero

Reviewers

Yifan Cai

• Created: 05/Feb/24 19:45
• Updated: 14/Feb/24 01:48
• Resolved: 14/Feb/24 01:47

Description

During bulk writes, Cassandra Analytics calculates the MD5 checksum of every SSTable it produces. During SSTable upload to Cassandra Sidecar, Cassandra Analytics includes the content-md5 header as part of the upload request. This information is used by Cassandra Sidecar to validate the integrity of the uploaded SSTable and prevent issues with bit flips and corrupted SSTables.

Recently, Cassandra Sidecar introduced support for additional checksum validations during SSTable upload. Notably the XXHash32 digest support was added which offers for more performant checksum calculations. This support now allows Cassandra Analytics to use a more efficient digest algorithm that is friendlier on the CPU usage of Sidecar and spark resources.

Issue Links

• links to: GitHub Pull Request #38