[FLINK-29298] LocalBufferPool request buffer from NetworkBufferPool hanging

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.17.0, 1.16.1
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Weijie Guo
• Reporter: Weijie Guo
• Created: 14/Sep/22 03:15
• Updated: 18/Mar/23 03:27
• Resolved: 13/Dec/22 04:30

Description

In the scenario where the buffer contention is fierce, sometimes the task hang can be observed. Through the thread dump information, we can found that the task thread is blocked by requestMemorySegmentBlocking forever. After investigating the dumped heap information, I found that the NetworkBufferPool actually has many buffers, but the LocalBufferPool is still unavailable and no buffer has been obtained.

By looking at the code, I am sure that this is a bug in thread race: when the task thread polled out the last buffer in LocalBufferPool and triggered the onGlobalPoolAvailable callback itself, it will skip this notification  (as currently the LocalBufferPool is available), which will cause the BufferPool to eventually become unavailable and will never register a callback to the NetworkBufferPool.

The conditions for triggering the problem are relatively strict, but I have found a stable way to reproduce it, I will try to fix and verify this problem.

Issue Links

• causes: FLINK-29419 HybridShuffleITCase.testHybridFullExchangesRestart hangs
• causes: FLINK-29923 Hybrid Shuffle may face deadlock when running a task need to execute big size data
• links to: GitHub Pull Request #20924