[FLINK-22993] CompactFileWriter won't emit EndCheckpoint with Long.MAX_VALUE checkpointId

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.14.0, 1.13.2
• Component/s: Connectors / FileSystem, Connectors / Hive
• Labels: pull-request-available
• Assignee: luoyuxia
• Reporter: luoyuxia
• Created: 15/Jun/21 06:53
• Updated: 23/Sep/21 17:26
• Resolved: 21/Jun/21 06:26

Description

CompactFileWriter won't emit EndCheckpoint with Long.MAX_VALUE checkpointId even though the inputs end. 

It will cause data lose. For example, after completing checkpoint 5, CompactFileWriter write some file, then the inputs end, but it won't emit EndCheckpoint  with Long.MAX_VALUE checkpointId, so the downstream operator won't do the compaction for the files which means these files are invisible.

Issue Links

• duplicates: FLINK-22073 ParquetFileCompactionITCase fails on Azure
• links to: GitHub Pull Request #16161