[FLINK-18738] Revisit resource management model for python processes.

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Resolved
• Fix Version/s: 1.12.0
• Component/s: API / Python, Runtime / Coordination
• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 28/Jul/20 07:55
• Updated: 27/Apr/21 18:06
• Resolved: 29/Oct/20 08:27

Description

This ticket is for tracking the effort towards a proper long-term resource management model for python processes.

In FLINK-17923, we run into problems due to python processes are not well integrate with the task manager resource management mechanism. A temporal workaround has been merged for release-1.11.

Issue Links

• is fixed by: FLINK-19177 FLIP-141: Intra-Slot Managed Memory Sharing
• relates to: FLINK-17923 It will throw MemoryAllocationException if rocksdb statebackend and Python UDF are used in the same slot