[FLINK-20269] The flush happens too frequent in SavepoinV2Serializer

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.9.0
• Component/s: Runtime / Checkpointing
• Assignee: Unassigned
• Reporter: Jiayi Liao
• Created: 21/Nov/20 09:33
• Updated: 22/Nov/20 10:09
• Resolved: 22/Nov/20 10:09

Description

The reason I notice this is, I find that the metadata's persistence can be very slow (but the states' uploading process works fine) when the network is unstable, and almost every time I dump the stack of the process, the bottleneck happens on Hdfs client waiting for Datanodes' ack during metadata's persistance.

 

I wonder, is it really necessary to flush the stream after every StreamStateHandle's serialization?

Issue Links

• duplicates: FLINK-19325 Optimize the consumed time for checkpoint completion