[FLINK-31293] Request memory segment from LocalBufferPool may hanging forever.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.17.0, 1.16.1, 1.18.0
• Fix Version/s: 1.16.2, 1.18.0, 1.17.1
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Weijie Guo
• Reporter: Weijie Guo
• Created: 02/Mar/23 04:21
• Updated: 04/Apr/23 02:37
• Resolved: 04/Apr/23 02:36

Description

In our TPC-DS test, we found that in the case of fierce competition in network memory, some tasks may hanging forever.

From the thread dump information, we can see that the task is waiting for the LocalBufferPool to become available. It is strange that other tasks have finished and released network memory already. Undoubtedly, this is an unexpected behavior, which implies that there must be a bug in the LocalBufferPool or NetworkBufferPool.

By dumping the heap memory, we can find a strange phenomenon that there are available buffers in the LocalBufferPool, but it was considered to be un-available. Another thing to note is that it now holds an overdraft buffer.

TL;DR: This problem occurred in multi-thread race related to the introduction of overdraft buffer.

Suppose we have two threads, called A and B. For simplicity, LocalBufferPool is called LocalPool and NetworkBufferPool is called GlobalPool.

Thread A continuously request buffers blocking from the LocalPool.
Thread B continuously return buffers to GlobalPool.

1. If thread A takes the last available buffer of LocalPool, but GlobalPool does not have a buffer at this time, it will register a callback function with GlobalPool.
2. Thread B returns one buffer to GlobalPool, but has not started to trigger the callback.
3. Thread A continues to request buffer. Because the availableMemorySegments of LocalPool is empty, it requests the overdraftBuffer instead. But there is already a buffer in the GlobalPool, it successfully gets the buffer.
4. Thread B triggers the callback. Since there is no buffer in GlobalPool now, the callback is re-registered.
5. Thread A continues to request buffer. Because there is no buffer in GlobalPool, it will block on CompletableFuture#get.
6. Thread B continues to return a buffer and triggers the recently registered callback. As a result, LocalPool puts the buffer into availableMemorySegments. Unfortunately, the current logic of shouldBeAvailable method is: if there is an overdraft buffer, LocalPool is considered as un-available.
7. Thread A will keep blocking forever.

Issue Links

• fixes: FLINK-31104 TPC-DS test timed out in query 36
• is caused by: FLINK-26762 Add the overdraft buffer in BufferPool to reduce unaligned checkpoint being blocked
• links to: GitHub Pull Request #22084