[FLINK-16406] Increase default value for JVM Metaspace to minimise its OutOfMemoryError

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.10.0
• Fix Version/s: 1.10.1, 1.11.0
• Component/s: Runtime / Configuration, Runtime / Task
• Labels: pull-request-available
• Assignee: Andrey Zagrebin
• Reporter: Andrey Zagrebin
• Created: 03/Mar/20 15:35
• Updated: 08/May/20 02:34
• Resolved: 12/Mar/20 09:36

Description

With FLIP-49 (FLINK-13980), we introduced a limit for JVM Metaspace ('taskmanager.memory.jvm-metaspace.size') when TM JVM process is started. It caused 'OutOfMemoryError: Metaspace' for some use cases after upgrading to the latest 1.10 version. In some cases, a real class loading leak has been discovered, like in FLINK-16142. Some users had to increase the default value to accommodate for their use cases (mostly from 96Mb to 256Mb).

While this limit was introduced to properly plan Flink resources, especially for container environment, and to detect class loading leaks, the user experience should be as smooth as possible. One way is provide good documentation for this change (FLINK-16278).

Another question is the sanity of the default value. It is still arguable what the default value should be (currently 96Mb). In general, the size depends on the use case (job user code, how many jobs are deployed in the cluster etc).

This issue tries to tackle this problem by firstly increasing it to 256Mb and overall default process size to 1728Mb in flink-conf.yaml to have no impact on default sizes of other memory components. We also want to poll which Metaspace setting resolved the OutOfMemoryError. Please, if you encountered this problem, report here any relevant specifics of your job and your Metaspace size if there was no class loading leak.

Issue Links

• relates to: FLINK-16142 Memory Leak causes Metaspace OOM error on repeated job submission
• relates to: FLINK-13980 FLIP-49 Unified Memory Configuration for TaskExecutors
• links to: GitHub Pull Request #11300