[FLINK-13752] TaskDeploymentDescriptor cannot be recycled by GC due to referenced by an anonymous function

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination
• Assignee: Yun Gao
• Reporter: Yun Gao
• Created: 16/Aug/19 16:08
• Updated: 19/Aug/19 23:10
• Resolved: 19/Aug/19 11:44

Description

When comparing the 1.8 and 1.9.0-rc2 on a test streaming job, we found that the performance on 1.9.0-rc2 is much lower than that of 1.8. By comparing the two versions, we found that the count of Full GC of TaskExecutor process on 1.9.0-rc2 is much more than that on 1.8.

A further analysis found that the difference is due to in TaskExecutor#setupResultPartitionBookkeeping, the anonymous function in taskTermimationWithResourceCleanFuture has referenced the TaskDeploymentDescriptor, since this function will be kept till the task is terminated,  TaskDeploymentDescriptor will also be kept referenced in the closure and cannot be recycled by GC. In this job, TaskDeploymentDescriptor of some tasks are as large as 10M, and the total heap is about 113M, thus the kept TaskDeploymentDescriptors will cause relatively large impact on GC and performance.

Issue Links

• links to: Github PR #9480