[FLINK-31610] Refactoring of LocalBufferPool

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.17.0
• Component/s: Runtime / Network
• Assignee: Unassigned
• Reporter: Anton Kalashnikov
• Created: 24/Mar/23 13:14
• Updated: 23/Apr/23 10:32

Description

FLINK-31293 bug highlighted the issue with the internal mutual consistency of different fields in LocalBufferPool. ex.:

• `numberOfRequestedOverdraftMemorySegments`
• `numberOfRequestedMemorySegments`
• `availableMemorySegment`
• `currentPoolSize`

Most of the problem was fixed already(I hope) but it is a good idea to reorganize the code in such a way that all invariants between all fields inside will be clearly determined and difficult to break.

As one example I can propose getting rid of numberOfRequestedOverdraftMemorySegments and using existing numberOfRequestedMemorySegments instead. That means:

• the pool will be available when `!availableMemorySegments.isEmpty() && unavailableSubpartitionsCount == 0`
• we don't request a new `ordinary` buffer when `numberOfRequestedMemorySegments >=  currentPoolSize` but we request the overdraft buffer instead
• `setNumBuffers` should work automatically without any changes

I think we can come up with a couple of such improvements to simplify the code.

Issue Links

• incorporates: FLINK-31764 Get rid of numberOfRequestedOverdraftMemorySegments in LocalBufferPool
• is related to: FLINK-31763 Convert requested buffers to overdraft buffers when pool size is decreased