Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.1.3
-
None
Description
Currently if BucketingSink receives no data after a checkpoint and then a notification about a previous checkpoint arrives, it clears its state. This can
lead to not committing valid data about intermediate checkpoints for whom
a notification has not arrived yet. As a simple sequence that illustrates the
problem:
-> input data
-> snapshot(0)
-> input data
-> snapshot(1)
-> no data
-> notifyCheckpointComplete(0)
the last will clear the state of the Sink without committing as final the data
that arrived for checkpoint 1.
Attachments
Issue Links
- links to