[FLINK-36394] Add jvm metaspaceSize opts to jobManager & taskManager.

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.0, 1.17.2
• Fix Version/s: 1.16.0, 1.17.0
• Component/s: API / Core
• Assignee: Unassigned
• Reporter: JieTan
• Created: 27/Sep/24 03:28
• Updated: 16/Oct/24 03:14

Description

We should set the size of the Init JVM Metaspace GC threadhold opts to a sane default, like -XX:MetaspaceSize=128m.

When a Flink job starts, it needs to load class metadata information, including class structure information, method information, and field information. As a result, the JVM frequently performs FullGC. Sometimes, the Metadata GC (Full GC) is 3.7s long before the Akka System is started.

 

2024-07-23T20:08:40.554+0800: 1.801: [Full GC (Metadata GC Threshold) 2024-07-23T20:08:40.554+0800: 1.801: [Tenured: 0K->14029K(966656K), 0.0336205 secs] 108273K->14029K(1401664K), [Metaspace: 20594K->20594K(1067008K)], 0.0337494 secs] [Times: user=0.02 sys=0.00, real=0.04 secs] 
2024-07-23T20:08:42.052+0800: 3.300: [Full GC (Metadata GC Threshold) 2024-07-23T20:08:42.052+0800: 3.300: [Tenured: 14029K->28979K(966656K), 0.0503497 secs] 184494K->28979K(1401664K), [Metaspace: 34384K->34384K(1079296K)], 0.0505108 secs] [Times: user=0.04 sys=0.00, real=0.05 secs] 
2024-07-23T20:08:45.064+0800: 6.312: [Full GC (Metadata GC Threshold) 2024-07-23T20:08:45.064+0800: 6.312: [Tenured: 28979K->54550K(966656K), 0.0876744 secs] 248437K->54550K(1401664K), [Metaspace: 57015K->57015K(1099776K)], 0.0879053 secs] [Times: user=0.07 sys=0.01, real=0.09 secs]  

 

The final Metaspace size takes up 100MB+. 

Solution: 

If we configure Init JVM Metaspace GC threadhold opts size by default. Maybe it can be set to the same value as `jobmanager.memory.jvm-metaspace.size`. As a result, Flink job Metaspace FullGC reduced.

 

OpenJDK 64-Bit Server VM (25.362-b09) for linux-amd64 JRE (1.8.0_362-ByteOpenJDK-b09), built on Feb 20 2023 09:42:31 by "root" with gcc 8.3.0
Memory: 4k page, physical 4018480k(137636k free), swap 0k(0k free)
CommandLine flags: -XX:CompressedClassSpaceSize=260046848 -XX:GCLogFileSize=104857600 -XX:InitialHeapSize=3368026112 -XX:MaxHeapSize=3368026112 -XX:MaxMetaspaceSize=268435456 -XX:MetaspaceSize=134217728 -XX:NumberOfGCLogFiles=5 -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseGCLogFileRotation