[FLINK-2491] Support Checkpoints After Tasks Finished

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 0.10.0
• Fix Version/s: 1.14.0
• Component/s: Runtime / Checkpointing
• Assignee: Yun Gao
• Reporter: Robert Metzger
• Created: 06/Aug/15 09:50
• Updated: 12/Apr/22 13:02
• Resolved: 21/Sep/21 09:18

Description

While implementing a test case for the Kafka Consumer, I came across the following bug:

Consider the following topology, with the operator parallelism in parentheses:

Source (2) --> Sink (1).

In this setup, the snapshotState() method is called on the source, but not on the Sink.
The sink receives the generated data.
only one of the two sources is generating data.

I've implemented a test case for this, you can find it here: https://github.com/rmetzger/flink/blob/para_checkpoint_bug/flink-tests/src/test/java/org/apache/flink/test/checkpointing/ParallelismChangeCheckpoinedITCase.java

Issue Links

• causes: FLINK-23829 SavepointITCase JVM crash on azure
• causes: FLINK-24161 Can not stop the job with savepoint while a task is finishing
• is duplicated by: FLINK-13270 CheckpointCoordinator abort the checkpoint
• is duplicated by: FLINK-9640 Checkpointing is aways aborted if any task has been finished
• relates to: FLINK-21839 SinkFunction snapshotState don't snapshot all data when trigger a stop-with-drain savepoint
• relates to: FLINK-27202 NullPointerException on stop-with-savepoint with AsyncWaitOperator followed by FlinkKafkaProducer
• relates to: FLINK-22764 Disable Values Source node in streaming plan optimization
• relates to: FLINK-23883 (1.15) Follow up tasks and code improvements for FLIP-147
• mentioned in: Page Loading...