[FLINK-13205] Checkpoints/savepoints injection has loose ordering properties when a stop-with-savepoint is triggered

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Checkpointing
• Labels: pull-request-available
• Assignee: Alex
• Reporter: Alex
• Created: 11/Jul/19 07:46
• Updated: 15/Jul/19 12:40
• Resolved: 15/Jul/19 12:38

Description

When a stop-with-savepoint is triggered at a source task, the task's dispatcher (Task.asyncCallDispatcher)'s thread pool is extended (from single-threaded, it becomes multi-threaded).

This leads to a race of applying consequent checkpoints/savepoints from dispatcher's queue at the same time and checkpoints/savepoints would be not strictly ordered in the event stream.

As the result, checkpoints/savepoints that injected later than they should, may be "silently subsumed": potentially, they would be ignored and won't be reported to checkpoint coordinator.

Proposed solution:

Revert Task.asyncCallDispatcher behavior to be single-threaded.
For stop-with-savepoint feature, the dispatcher's thread that performs the synchronous savepoint doesn't need to be blocking and StreamTask.finishTask() invocation can be delegated to StreamTask.notifyCheckpointComplete().

Note: imo, the issue described here is not critical, but the proposed change should simplify implementation. This ticket can be considered as enhancement.

Issue Links

• links to: GitHub Pull Request #9021