[FLINK-25511] Pre-emptively uploaded changelog not discarded up if materialized before checkpoint

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / Checkpointing, Runtime / State Backends
• Labels: pull-request-available
• Assignee: Roman Khachatryan
• Reporter: Yuan Mei
• Created: 04/Jan/22 08:08
• Updated: 23/Aug/22 09:41
• Resolved: 28/Apr/22 18:26

Description

This is related to Garbage Collection. 

The problem is more severe than FLINK-25512 because there is leftover after materialization truncation each time (if pre-emptive uploads are enabled).

However, the probability can still be not high because of grouping of backends into a single file (so only if ALL the backends on TM materialize semi-simultaneously then it happen).

Issue Links

• causes: FLINK-27556 Performance regression in checkpointSingleInput.UNALIGNED on 29.04.2022
• causes: FLINK-29082 Clean-up Leftovers for changelog pre-uploading files after failover
• is related to: FLINK-23139 State ownership: track and discard private state (registry+changelog)
• is related to: FLINK-24852 Cleanup of Orphaned Incremental State Artifacts
• links to: GitHub Pull Request #19550