[FLINK-32881] Client supports making savepoints in detach mode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.19.0
• Fix Version/s: 1.19.0
• Component/s: Client / Job Submission, Runtime / Checkpointing
• Labels: detach-savepoint pull-request-available
• Flags: Important

Language

Java

• Assignee: Renxiang Zhou
• Reporter: Renxiang Zhou
• Created: 16/Aug/23 09:28
• Updated: 27/Dec/23 02:50
• Resolved: 27/Dec/23 02:50

Description

When triggering a savepoint using the command-line tool, the client needs to wait for the job to finish creating the savepoint before it can exit. For jobs with large state, the savepoint creation process can be time-consuming, leading to the following problems:

1. Platform users may need to manage thousands of Flink tasks on a single client machine. With the current savepoint triggering mode, all savepoint creation threads on that machine have to wait for the job to finish the snapshot, resulting in significant resource waste;
2. If the savepoint producing time exceeds the client's timeout duration, the client will throw a timeout exception and report that the triggering savepoint process fails. Since different jobs have varying savepoint durations, it is difficult to adjust the timeout parameter on the client side.

Therefore, we propose adding a detach mode to trigger savepoints on the client side, just similar to the detach mode behavior when submitting jobs. Here are some specific details:

1. The savepoint UUID will be generated on the client side. After successfully triggering the savepoint, the client immediately returns the UUID information and exits.
2. Add a "dump-pending-savepoints" API that allows the client to check whether the triggered savepoint has been successfully created.

By implementing these changes, the client can detach from the savepoint creation process, reducing resource waste, and providing a way to check the status of savepoint creation.

Issue Links

• links to: GitHub Pull Request #23253