[FLINK-24491] ExecutionGraphInfo may not be archived when the dispatcher terminates

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.6, 1.14.4, 1.15.0
• Fix Version/s: 1.14.5, 1.15.1, 1.16.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Matthias Pohl
• Reporter: Zhilong Hong
• Created: 09/Oct/21 07:04
• Updated: 03/Aug/22 11:48
• Resolved: 08/Jun/22 08:58

Description

When a job finishes, its JobManagerRunnerResult will be processed in the callback of Dispatcher#runJob. In the callback, ExecutionGraphInfo will be archived by HistoryServerArchivist asynchronously. However, the CompletableFuture of the archiving is ignored. The job may be removed before the archiving is finished. For the batch job running in the per-job/application mode, the dispatcher will terminate itself once the job is finished. In this case, ExecutionGraphInfo may not be archived when the dispatcher terminates.

If the ExecutionGraphInfo is lost, users are not able to know whether the batch job is finished normally or not. They have to refer to the logs for the result.

The session mode is not affected, since the dispatcher won't terminate itself once the job is finished. The HistoryServerArchivist gets enough time to archive the ExcutionGraphInfo.

Issue Links

• is duplicated by: FLINK-28531 Shutdown cluster after history server archive finished
• relates to: FLINK-26772 Application and Job Mode does not wait for job cleanup during shutdown
• Testing discovered: FLINK-26976 HistoryServer archiving is not idempotent
• Testing discovered: FLINK-26984 HistoryServer archiving is not retried
• links to: 1.14 BP GitHub Pull Request
• links to: 1.15 BP GitHub Pull Request
• links to: GitHub Pull Request #19275
• links to: GitHub Pull Request #19591