[FLINK-24377] TM resource may not be properly released after heartbeat timeout

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0, 1.13.2
• Fix Version/s: 1.13.3, 1.14.3, 1.15.0
• Component/s: Deployment / Kubernetes, Deployment / YARN, Runtime / Coordination
• Labels: pull-request-available
• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 26/Sep/21 08:08
• Updated: 15/Dec/21 01:44
• Resolved: 29/Sep/21 01:36

Description

In native k8s and yarn deploy modes, RM disconnects a TM when its heartbeat times out. However, it does not actively release the pod / container of that TM. The releasing of pod / container relies on the TM to terminate itself after failing to re-register to the RM.

In some rare conditions, the TM process may not terminate and hang out for long time. In such cases, k8s / yarn sees the process running, thus will not release the pod / container. Neither will Flink's resource manager. Consequently, the resource is leaked until the entire application is terminated.

To fix this, we should make ActiveResourceManager to actively release the resource to K8s / Yarn after a TM heartbeat timeout.

Issue Links

• links to: GitHub Pull Request #17362