[FLINK-33879] Hybrid Shuffle may stop working for a while during redistribution

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.19.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Jiang Xin
• Reporter: Jiang Xin
• Created: 19/Dec/23 10:07
• Updated: 15/Jan/24 02:18
• Resolved: 15/Jan/24 02:18

Description

Currently, the Hybrid Shuffle can work with the memory tier and disk tier together, however, in the following scenario the result partition would stop working.

Suppose we have a shuffle task with 2 sub-partitions. The LocalBufferPool has 15 buffers, the memory tier can use at most 15-(2*(2+1)+1) = 8 buffers according to `TieredStorageMemoryManagerImpl#getMaxNonReclaimableBuffers`. If the memory tier uses up all 8 buffers and the input channel consumes them very slowly because of problems, e.g. unstable network, the disk tier can still work with 1 reserved buffer. However, if a redistribution happens now and the pool size is decreased to less than 8, then the BufferAccumulator can not request buffers anymore, and thus the result partition stops working until the buffers in the memory tier are recycled.

The purpose is to make the result partition still work with the disk tier and write the shuffle data to disk so that once the input channel is ready, the data on the disk can be consumed immediately.

Issue Links

• links to: GitHub Pull Request #23957