Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Invalid
-
None
-
None
-
None
Description
Right now, many output connectors exist in two variants: OutputFormat and StreamSink.
Wrapping an output format to a stream sink is not quite possible, because output formats cannot be cleanly integrated into checkpointing. For example, they have no hook to ensure that data is written/persisted at the time of a checkpoint.
We should try and see if we can make both one, or at least make the OutputFormats such that they can be wrapped into a stream sink and behave well with respect to checkpointing.