[FLINK-17820] Memory threshold is ignored for channel state

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.0
• Component/s: Runtime / Checkpointing, Runtime / Task
• Labels: pull-request-available
• Assignee: Roman Khachatryan
• Reporter: Roman Khachatryan
• Created: 19/May/20 21:27
• Updated: 16/Oct/20 10:56
• Resolved: 27/May/20 15:35

Description

Config parameter state.backend.fs.memory-threshold is ignored for channel state. Causing each subtask to have a file per checkpoint. Regardless of the size of channel state (of this subtask).

This also causes slow cleanup and delays the next checkpoint.

 

The problem is that ChannelStateCheckpointWriter.finishWriteAndResult calls flush(); which actually flushes the data on disk.

 

From FSDataOutputStream.flush Javadoc:

A completed flush does not mean that the data is necessarily persistent. Data persistence can is only assumed after calls to close() or sync().

 

Possible solutions:

1. not to flush in ChannelStateCheckpointWriter.finishWriteAndResult (which can lead to data loss in a wrapping stream).

2. change FsCheckpointStateOutputStream.flush behavior

3. wrap FsCheckpointStateOutputStream to prevent flush}}{{

Issue Links

• causes: FLINK-17986 Erroneous check in FsCheckpointStateOutputStream#write(int)
• links to: GitHub Pull Request #12258
• links to: GitHub Pull Request #12332
• links to: GitHub Pull Request #12351