[FLINK-24864] Release TaskManagerJobMetricGroup with the last slot rather than task

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Runtime / Metrics, Runtime / State Backends
• Labels: pull-request-available
• Release Note: TM-per-job metrics are now released upon last slot release rather than task, i.e. may stay on TM longer.
• Assignee: Roman Khachatryan
• Reporter: Roman Khachatryan
• Created: 10/Nov/21 14:15
• Updated: 16/Nov/21 08:24
• Resolved: 16/Nov/21 08:24

Description

// Beware that this ticket FLINK-24864 clones FLINK-23486.

 

FLINK-23486 adds metrics for Changelog Uploader.

Since that uploader is shared among tasks of the same job on a TM, the right level for metrics is TaskManagerJobMetricGroup (see design doc).

 

However, the lifecycle of TaskManagerJobMetricGroup differs from that of StateChangelogStorage: the former is released on last task unregister; the latter - on last job slot release.

This causes problem that an old TMJMG can be used by Storage.

 

Releasing Storage on last task unregister has some disadvantages, including overhead, thread-safety and logical inconsistency.

Releasing TMJMG on last slot release doesn't have major disadvantages.

This ticket proposes the former change.

Issue Links

• blocks: FLINK-23486 Add metrics for the Changelog uploader
• is a clone of: FLINK-23486 Add metrics for the Changelog uploader
• links to: GitHub Pull Request #17387