[FLINK-28248] Metaspace memory is leaking when repeatedly submitting Beam batch pipelines via the REST API

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.4
• Component/s: API / Core
• Assignee: Unassigned
• Reporter: Arkadiusz Gasinski
• Created: 24/Jun/22 12:49
• Updated: 06/Jul/22 06:55

Description

We have a Flink cluster running on k8s/OpenShift in session mode running our Apache Beam pipelines. Some of these pipelines are streaming pipelines and run continuously; some are batch pipelines submitted periodically whenever there is a load to be processed.

We believe that the batch pipelines cause the issue. We submit 1 to several batch jobs every 5 minutes. For each job, a new instance of the ChildFirstClassLoader is instantiated and it looks like they are not closed properly after the job finishes.

Attached is the screenshot from the Eclipse memory analyzer - from the Leak Suspects report. When the heap dump was captured, there were 2 streaming and several batch jobs running plus over 100 finished batch jobs.

In our current setup, we allocate 8GB for the metaspace:

 

And the top components from the mem analyzer: