[BEAM-4775] JobService should support returning metrics

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: beam-model
• Assignee: Lukasz Gajowy
• Reporter: Eugene Kirpichov
• Created: 12/Jul/18 21:46
• Updated: 16/May/20 14:12
• Resolved: 18/Nov/19 22:15

Description

Design doc: https://s.apache.org/get-metrics-api.

Further discussion is ongoing on this doc.

We want to report job metrics back to the portability harness from the runner harness, for displaying to users.

Relevant PRs in flight:

Ready for Review:

• #8022: correct the Job RPC protos from #8018.

Iterating / Discussing:

• #7971: Flink portable metrics: get ptransform from MonitoringInfo, not stage name
  • this is a simpler, Flink-specific PR that is basically duplicated inside each of the following two, so may be worth trying to merge in first
• #7915: use MonitoringInfo data model in Java SDK metrics
• #7868: MonitoringInfo URN tweaks

Merged

• #8018: add job metrics RPC protos
• #7867: key MetricResult by a MetricKey
• #7938: move MonitoringInfo protos to model/pipeline module
• #7883: Add MetricQueryResults.allMetrics() helper
• #7866: move function helpers from fn-harness to sdks/java/core
• #7890: consolidate MetricResult implementations

Closed

• #7934: job metrics RPC + SDK support
• #7876: Clean up metric protos; support integer distributions, gauges

Issue Links

• is depended upon by: BEAM-4776 Java PortableRunner should support metrics
• supercedes: BEAM-1866 Fn API support for Metrics
• links to: GitHub Pull Request #7641
• links to: GitHub Pull Request #7656
• links to: GitHub Pull Request #7823
• links to: GitHub Pull Request #7866
• links to: GitHub Pull Request #7867
• links to: GitHub Pull Request #7868
• links to: GitHub Pull Request #7876
• links to: GitHub Pull Request #7934
• links to: GitHub Pull Request #7938
• links to: GitHub Pull Request #8018
• links to: GitHub Pull Request #8022
• links to: GitHub Pull Request #9020
• links to: GitHub Pull Request #9843