[CASSANDRA-19563] [Analytics] Support bulk write via S3

• Type: New Feature
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: NA
• Component/s: Analytics Library
• Change Category: Semantic
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra-analytics/commit/aea798dc7e517af520a403d4d86f3bc6bed65092

Test and Documentation Plan

ci; unit + integration

• Assignee: Yifan Cai
• Reporter: Yifan Cai

Authors

Yifan Cai

Reviewers

Francisco Guerrero

• Created: 16/Apr/24 18:30
• Updated: 22/Apr/24 22:46
• Resolved: 22/Apr/24 22:46

Description

I would like to propose a new write option in Cassandra Analytics to bulk write SSTables via S3, in addition to the previously-implemented "direct upload to all sidecars" (now known as the "Direct" transport).
The new write option, now being implemented, is the "S3_COMPAT" transport, which allows the job to upload the generated SSTables to an S3-compatible storage system, and then inform the Cassandra Sidecar that those files are available for download & commit.

Additionally, a plug-in system was added to allow communications between custom transport hooks and the job, so the custom hook can provide updated credentials and out-of-band status updates on S3-related issues.

Issue Links

• links to: GitHub Pull Request #53