[FLINK-19177] FLIP-141: Intra-Slot Managed Memory Sharing

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.12.0
• Component/s: Runtime / Coordination
• Labels: Umbrella

Release Note

The configuration python.fn-execution.buffer.memory.size and python.fn-execution.framework.memory.size have been removed and so will not take effect any more. Besides, the default value of python.fn-execution.memory.managed has been changed to true and so managed memory will be used by default for Python workers. In cases where Python UDFs are used together with RocksDB state backend in streaming or built-in batch algorithms in batch, the user can control how managed memory should be shared between data processing (RocksDB state backend or batch algorithms) and Python, by overwriting [managed memory consumer weights]({% link ops/mem_setup_tm.md %}#consumer-weights).

• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 10/Sep/20 02:10
• Updated: 14/Jan/21 09:20
• Resolved: 17/Nov/20 08:57

Description

This is the umbrella ticket of FLIP-141: Intra-Slot Managed Memory Sharing.

FLIP-53 introduced a fraction based approach for sharing managed memory within a slot. This approach needs to be extended as python operators, which also use managed memory, are introduced. This FLIP proposes a design for extending intra-slot managed memory sharing for python operators and other potential future managed memory use cases.

Issue Links

• fixes: FLINK-18738 Revisit resource management model for python processes.
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Introduce the memory weights configuration option and interfaces: Closed
• Implement the managed memory fraction calculation logic: Closed
• Make RocksDB respect the calculated managed memory fraction: Closed
• Make python processes respect the calculated managed memory fraction: Closed
• Update document for intra-slot managed memory sharing: Closed