[BEAM-9997] Improve step names in portable runners

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-spark
• Labels: portability-spark
• Assignee: Unassigned
• Reporter: Kyle Weaver
• Created: 14/May/20 17:57
• Updated: 04/Jun/22 15:41

Description

Step names are currently inconsistent between runners. This is probably unavoidable due to fusion, but we should design a system that is more consistent between runners so that most metric queries against the Fn API runner can also be used with the portable Flink and Spark runners.

Spark Runner:
MetricKey(step=ref_AppliedPTransform_count1_17, metric=MetricName(namespace=ns, name=counter), labels={}): 2
MetricKey(step=ref_AppliedPTransform_count2_18, metric=MetricName(namespace=ns, name=counter), labels={}): 4
...

Fn API Runner:
MetricKey(step=count1, metric=MetricName(namespace=ns, name=counter), labels={}): 2,
MetricKey(step=count2, metric=MetricName(namespace=ns, name=counter), labels={}): 4

Issue Links

• is related to: BEAM-9996 Flink should not shadow FnApiRunnerTest.test_metrics
• relates to: BEAM-7219 Spark portable runner: support Spark metrics