[BEAM-1659] Improve UnboundedSourceShard structure

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-direct
• Labels: newbie starter
• Assignee: Unassigned
• Reporter: Thomas Groh
• Created: 08/Mar/17 19:34
• Updated: 03/Jun/22 16:22

Description

Use the 'OneOf' of "Active Reader" and "Optional<Checkpoint>", to make it obvious how to obtain and use a reader.

Maintain an ordered list of checkpoints that have not been finalized for a shard. Checkpoints that can be resumed from can also be finalized, but otherwise the two are unrelated. This also improves ease of never multi-finalizing a checkpoint. This gets more complicated with retries, but must be considered with the associated work required to make UnboundedReadEvaluatorFactory handle retries properly.

Issue Links

• relates to: BEAM-1656 DirectRunner should not call finalize twice in UnboundedSourceExecutorFactory