[FLINK-29615] MetricStore does not remove metrics of nonexistent subtasks when adaptive scheduler lowers job parallelism

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0, 1.16.0
• Fix Version/s: 1.17.0, 1.16.1
• Component/s: Runtime / Metrics, Runtime / REST
• Labels: pull-request-available
• Assignee: Zhanghao Chen
• Reporter: Zhanghao Chen
• Created: 13/Oct/22 06:45
• Updated: 21/Oct/22 02:20
• Resolved: 21/Oct/22 02:20

Description

We are exploring autoscaling Flink with Reactive mode using metrics from Flink REST for guidance, and found that the metrics are not correctly updated.

 

Problem

MetricStore does not remove metrics of nonexistent subtasks when adaptive scheduler lowers job parallelism (aka, num of subtasks decreases) and users will see metrics of nonexistent subtasks on Web UI (e.g. the task backpressure page) or REST API response. It causes confusion and occupies extra memory.

 

Proposed Solution

Thanks to FLINK-29132 & FLINK-28588,  Flink will now update current execution attempts when updating metrics. Since the active subtask info is included in the current execution attempt info, we are able to retain active subtasks using the current execution attempt info.

 

Issue Links

• links to: GitHub Pull Request #21051