Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Implemented
-
None
-
None
Description
Some operators maintain some small transient state that may be inefficient to checkpoint, especially when it would need to be checkpointed also in a re-scalable way.
An example are opportunistic pre-aggregation operators, which have small the pre-aggregation state that is frequently flushed downstream.
Rather that persisting that state in a checkpoint, it can make sense to flush the data downstream upon a checkpoint, to let it be part of the downstream operator's state.
This feature is sensitive, because flushing state has a clean implication on the downstream operator's checkpoint alignment. However, used with care, and with the new back-pressure-based checkpoint alignment, this feature can be very useful.
Because it is sensitive, I suggest to make this only an internal feature (accessible to operators) and NOT expose it in the public API at this point.
Attachments
Issue Links
- links to
