[OAK-10634] Generalise Split segment store persistence

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: segment-tar
• Assignee: Andrei Dulceanu
• Reporter: Andrei Dulceanu
• Created: 26/Jan/24 12:28
• Updated: 26/Jan/24 12:28

Description

Introduced with OAK-7735,

Split segment store persistence is a proxy layer that can be used between the SegmentMK and the actual segment store persistence. It's configured with two backends: read-only and read-write. It delegates all the read requests to the read-only backend, but any write request (eg. creating a new segment) is delegated to the read-write backend.

It would be nice to have a generalisation of Split segment store service allowing the composition of multiple read-only backends and a single read-write backend.

The main advantage is that we can have multiple segment stores in the cloud (using the oak-segment-azure), shared amongst many Oak instances, starting dynamically.

The implementation remembers the state (tar file list, last journal entry) of the read-only backends during its initialisation, so the read-only backend can be used by a different instance, as long as it only appends new segments (e.g. no compaction).