[FLINK-13980] FLIP-49 Unified Memory Configuration for TaskExecutors

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.10.0
• Component/s: Runtime / Configuration, Runtime / Coordination
• Labels: Umbrella

Release Note

With FLINK-13980, a new memory model has been introduced for the task executor. New configuration options have been introduced to control the memory consumption of the task executor process. This affects all types of deployments: standalone, YARN, Mesos, and the new active Kubernetes integration. The memory model of the job manager process has not been changed yet but it is planned to be updated as well.

If you try to reuse your previous Flink configuration without any adjustments, the new memory model can result in differently computed memory parameters for the JVM and, thus, performance changes.

Please note that the following options have been removed and have no effect anymore: "taskmanager.memory.fraction", "taskmanager.memory.off-heap", "taskmanager.memory.preallocate".

The following options, if used, are interpreted as other new options in order to maintain backwards compatibility where it makes sense: "taskmanager.heap.size", "taskmanager.memory.size", "taskmanager.network.memory.min", "taskmanager.network.memory.max", "taskmanager.network.memory.fraction".


The container cut-off configuration options, `containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`, have no effect for task executor processes anymore but they still have the same semantics for the JobManager process.

More details, please refer to Flink user documents.

• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 06/Sep/19 02:51
• Updated: 16/Nov/20 14:18
• Resolved: 10/Feb/20 09:40

Description

This is the umbrella issue of 'FLIP-49: Unified Memory Configuration for TaskExecutors'.

Issue Links

• causes: FLINK-15103 Performance regression on 3.12.2019 in various benchmarks
• causes: FLINK-15925 TaskExecutors don't work out-of-the-box on Windows
• contains: FLINK-9904 Allow users to control MaxDirectMemorySize
• duplicates: FLINK-12762 Memory configuration implementation
• is blocked by: FLINK-15763 Check or set necessary resource configuration options to defaults for local execution ignoring FLIP-49
• is duplicated by: FLINK-13477 Containerized TaskManager killed because of lack of memory overhead
• is related to: FLINK-14489 Check and make sure state backend fits into TaskExecutor memory configuration
• is related to: FLINK-15082 Mesos App Master does not respect taskmanager.memory.total-process.size
• is related to: FLINK-15300 Shuffle memory fraction sanity check does not account for its min/max limit
• is related to: FLINK-18153 flink-daemon does not detect errors if JVM crashes immediately
• is related to: FLINK-14883 Resource management on state backends
• is related to: FLINK-15774 Consider adding an explicit network memory size config option
• is related to: FLINK-15145 Tune default values for FLIP-49 TM memory configurations with real production jobs.
• is related to: FLINK-15597 Relax sanity check of JVM memory overhead to be within its min/max
• is related to: FLINK-14431 Update TaskManager's memory information to match its memory composition
• is related to: FLINK-15198 Remove deprecated mesos.resourcemanager.tasks.mem in 1.11
• is related to: FLINK-15517 Use back 'network' in 'shuffle' memory config option names
• is related to: FLINK-15530 Replace process memory with flink memory for TMs in default flink-conf.yaml
• is related to: FLINK-15692 Enable limiting RocksDB memory consumption by default
• is related to: FLINK-16406 Increase default value for JVM Metaspace to minimise its OutOfMemoryError
• is related to: FLINK-15989 Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message
• is related to: FLINK-16278 Document new settings for MaxMetaSpace in 1.10
• is related to: FLINK-16614 FLIP-116 Unified Memory Configuration for Job Manager
• is related to: FLINK-17503 Make memory configuration logging more user-friendly
• is related to: FLINK-20171 Improve error message for Flink process memory configuration
• relates to: FLINK-15143 Create document for FLIP-49 TM memory model and configuration guide
• relates to: FLINK-17973 Test memory configuration of Flink cluster
• relates to: FLINK-14894 HybridOffHeapUnsafeMemorySegmentTest#testByteBufferWrap failed on Travis
• relates to: FLINK-15144 Document performance related changes caused by FLIP-49
• links to: FLIP-49: Unified Memory Configuration for TaskExecutors

Sub-Tasks

• Introduce a switch for enabling the new task executor memory configurations: Resolved
• Implement memory calculation logics: Resolved
• Account for netty direct allocations in direct memory limit (Netty Shuffle): Closed
• Introduce framework off heap memory config: Resolved
• Account for netty direct allocations in direct memory limit (Queryable state): Closed
• Launch task executor with new memory calculation logics: Closed
• Align ResourceProfile/ResourceSpec fields with the new TaskExecutor memory setups.: Resolved
• Separate on-heap and off-heap managed memory pools: Closed
• Add memory chunk reservation API to MemoryManager: Closed
• Use unsafe memory for managed memory.: Resolved
• Adjust GC Cleaner for unsafe memory and Java 11: Closed
• Shrink the scope of MemoryManager from TaskExecutor to slot: Resolved
• Clean-up of legacy mode: Closed
• Remove on-heap managed memory: Closed