[FLINK-7781] Support simple on-demand metrics aggregation

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.4.0
• Component/s: Runtime / Metrics, Runtime / REST
• Assignee: Chesnay Schepler
• Reporter: Chesnay Schepler
• Created: 09/Oct/17 12:47
• Updated: 31/Oct/17 12:38
• Resolved: 31/Oct/17 12:37

Description

We should support aggregations (min, max, avg, sum) of metrics in the REST API. This is primarily about aggregating across subtasks, for example the number of incoming records across all subtasks.

This is useful for simple use-cases where a dedicated metrics backend is overkill, and will allow us to provide better metrics in the web UI (since we can expose these aggregated as well).

I propose to add a new query parameter "agg=[min,max,avg,sum]". As a start this parameter should only be used for task metrics. (This is simply the main use-case i have in mind)

The aggregation should (naturally) only work for numeric metrics.

We will need a HashSet of metrics that exist for subtasks of a given tasks that has to be updated in MetricStore#add.

All task metrics are either stored as

1. <subtask-index>.<metric> or
2. <subtask-index>.<operator-name>.<metric>.

If a user sends a request get=mymetric,agg=sum, only the metrics of the first kind are to be considered. Similarly, given a request get=myoperator.mymetric,agg=sum only metrics of the second kind are to be considered.

Ideally, the name of the aggregated metric (i.e. the original name without subtask index) is also contained in the list of available metrics.

Issue Links

• links to: GitHub Pull Request #4901