[FLINK-8946] TaskManager stop sending metrics after JobManager failover

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.2, 1.5.0, 1.6.0
• Fix Version/s: 1.4.3, 1.5.1, 1.6.0
• Component/s: Runtime / Coordination, Runtime / Metrics
• Assignee: vinoyang
• Reporter: Truong Duc Kien
• Created: 15/Mar/18 03:11
• Updated: 28/Feb/19 14:11
• Resolved: 12/Jun/18 12:32

Description

Running in Yarn-standalone mode, when the Job Manager performs a failover, all TaskManager that are inherited from the previous JobManager will not send metrics to the new JobManager and other registered metric reporters.

 

A cursory glance reveal that these line of code might be the cause

https://github.com/apache/flink/blob/a3478fdfa0f792104123fefbd9bdf01f5029de51/flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala#L1082-L1086

Perhap the TaskManager close its metrics group when disassociating JobManager, but not creating a new one on fail-over association ?

 

Issue Links

• links to: GitHub Pull Request #5869
• links to: GitHub Pull Request #6060