[FLINK-16408] Bind user code class loader to lifetime of a slot

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.2, 1.10.0
• Fix Version/s: 1.11.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

The user code class loader is being reused by the `TaskExecutor` as long as there is at least a single slot allocated for the respective job. This changes Flink's recovery behaviour slightly so that it will not reload static fields. The benefit is that this change drastically reduces pressure on the JVM's metaspace.

• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 03/Mar/20 18:04
• Updated: 02/Nov/20 08:57
• Resolved: 07/May/20 13:30

Description

In order to avoid class leaks due to creating multiple user code class loaders and loading class multiple times in a recovery case, I would suggest to bind the lifetime of a user code class loader to the lifetime of a slot. More precisely, the user code class loader should live at most as long as the slot which is using it.

Issue Links

• is duplicated by: FLINK-11205 Task Manager Metaspace Memory Leak
• is duplicated by: FLINK-10352 Bind Task's URLClassLoader to TaskExecutor's slot
• relates to: FLINK-19005 Document JDBC drivers as source of Metaspace leaks
• links to: GitHub Pull Request #11963