[FLINK-35557] MemoryManager only reserves memory per consumer type once

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.3, 1.17.2, 1.19.0, 1.18.1, 1.20.0
• Fix Version/s: 1.20.0, 1.19.2
• Component/s: Runtime / State Backends, Runtime / Task
• Assignee: Roman Khachatryan
• Reporter: Roman Khachatryan
• Created: 07/Jun/24 13:49
• Updated: 11/Jun/24 01:26

Description

1. In MemoryManager.getSharedMemoryResourceForManagedMemory we create a reserve function
2. The function decrements the available Slot memory and fails if there's not enough memory
3. We pass it to SharedResources.getOrAllocateSharedResource
4. In SharedResources.getOrAllocateSharedResource , we check if the resource (memory) was already reserved by some key (e.g. state-rocks-managed-memory)
5. If not, we create a new one and call the reserve function
6. If the resource was already reserved (not null), we do NOT reserve the memory again: https://github.com/apache/flink/blob/57869c11687e0053a242c90623779c0c7336cd33/flin[…]/main/java/org/apache/flink/runtime/memory/SharedResources.java

So there will be only one (first) memory reservation for rocksdb for example, no matter how many state backends in a slot are created. Meaning that managed memory limits are not applied