[FLINK-32598] Spill data from feedback edge to disk to avoid possible OOM

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: ml-2.4.0
• Component/s: Library / Machine Learning
• Assignee: Unassigned
• Reporter: Zhipeng Zhang
• Created: 17/Jul/23 07:29
• Updated: 17/Jul/23 07:43

Description

In Flink ML, we use feedback edge to implement the iteration module. Suppose the job topology is like `OpA -> HeadOperator -> OpB -> TailOperator`, then the basic process of each iteration is as follows:

• At the first iteration, HeadOperator takes the input from OpA and forward it to OpB.
• Later, OpB consumes the input from HeadOperator and forward the output to TailOperator.
• Finally, TailOperator puts the records into a memory message queue and HeadOperator consumes the message queue.

When the output from OpB contains many records and these records cannot be consumed soon, the message queue would grow big and finally lead to OOM.