[FLINK-26803] Merge small ChannelState file for Unaligned Checkpoint

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.17.0
• Component/s: Runtime / Checkpointing, Runtime / Network
• Labels: pull-request-available stale-assigned
• Assignee: Rui Fan
• Reporter: Rui Fan
• Created: 22/Mar/22 10:19
• Updated: 05/Jun/23 04:58
• Resolved: 23/Jan/23 15:48

Description

When making an unaligned checkpoint, the number of ChannelState files is TaskNumber * subtaskNumber. For high parallelism job, it writes too many small files. It causes high load for hdfs NN.

 

In our production, a job writes more than 50K small files for each Unaligned Checkpoint. Could we merge these files before write FileSystem? We can configure the maximum number of files each TM can write in a single Unaligned Checkpoint.

Issue Links

• causes: FLINK-31036 StateCheckpointedITCase timed out
• relates to: FLINK-28474 ChannelStateWriteResult may not fail after checkpoint abort
• links to: GitHub Pull Request #20151