[FLINK-33954] Large record may cause the hybrid shuffle hang

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / Network
• Assignee: Unassigned
• Reporter: Jiang Xin
• Created: 28/Dec/23 07:41
• Updated: 02/Jan/24 06:15

Description

In some cases, the job may hang when there are not enough buffers in the local buffer pool. For instance, the parallelism is 4, so the HashBufferAccumulator is used. The size of the local buffer pool can be 5, and at some point, 3 of all buffers are required by 3 subpartitions and are not finished, so only 2 buffers are left. If a record that is larger than 2 buffers comes, the program would hang at requesting buffers.