[FLINK-31763] Convert requested buffers to overdraft buffers when pool size is decreased

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.1
• Fix Version/s: 1.16.2, 1.18.0, 1.17.1
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Weijie Guo
• Reporter: Weijie Guo
• Created: 11/Apr/23 05:38
• Updated: 17/Apr/23 09:29
• Resolved: 17/Apr/23 09:29

Description

As we discussed in FLINK-31610, new buffers can be requested only when "numOfRequestedMemorySegments + numberOfRequestedOverdraftMemorySegments < poolSize + maxOverdraftBuffersPerGate".

Consider such a scenario, the CurrentPoolSize = 5, numOfRequestedMemorySegments = 7, maxOverdraftBuffersPerGate = 2. If numberOfRequestedOverdraftMemorySegments = 0, then 2 buffers can be requested now. 

We should convert numberOfRequestedMemorySegments to numberOfRequestedOverdraftMemorySegments when poolSize is decreased. Further more, we can changes the definition of overdraft buffer from static to dynamic: 

• When numberOfRequestedMemorySegments <= poolSize, all buffers are ordinary buffer
• When numberOfRequestedMemorySegments > poolSize, the `ordinary buffer size = poolSize`, and `the overdraft buffer size = numberOfRequestedMemorySegments - poolSize`

This allows us to remove numberOfRequestedOverdraftMemorySegments, which helps us simplify logic and maintain consistency.

 

Issue Links

• relates to: FLINK-31610 Refactoring of LocalBufferPool
• links to: GitHub Pull Request #22381