[FLINK-22008] writing metadata is not an atomic operation, we should add a commit logic

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.12.2
• Component/s: Runtime / Checkpointing
• Labels: auto-deprioritized-major pull-request-available stale-minor
• Assignee: Unassigned
• Reporter: xiaogang zhou
• Created: 29/Mar/21 07:35
• Updated: 28/Mar/22 08:38
• Resolved: 28/Mar/22 08:38

Description

writing metadata is not an atomic operation, some logic can cause there is a metadata file in the checkpoint dir, but the data is corrupted if the jobmanager crash while writing the metadata. 

 

So we should consider to add commit operation in the checkpoint storage location

Issue Links

• is depended upon by: FLINK-9043 Introduce a friendly way to resume the job from externalized checkpoints automatically
• is superceded by: FLINK-17808 Rename checkpoint meta file to "_metadata" until it has completed writing
• relates to: FLINK-17808 Rename checkpoint meta file to "_metadata" until it has completed writing
• links to: GitHub Pull Request #15409
• links to: GitHub Pull Request #15410