[BEAM-12401] Beam metrics are unreadable in Flink history server

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink
• Assignee: Unassigned
• Reporter: David Janicek
• Created: 25/May/21 13:16
• Updated: 04/Jun/22 20:43

Description

Beam metric values (payload) are not readable in Flink's web UI and Flink history server. Similar problem affected Spark history server and it was resolved in BEAM-10294.

Beam metrics are rendered by MetricsContainerStepMap.toString() method as JSON which is created from protobuffer defined in metrics.proto where metric's value is defined as bytes.

For example, metrics in Flink web UI used to look like:

MetricQueryResults(Counters(sets-by-key-bloomfilter/filter-by-bloom-filter/ParMultiDo(FilterByBloomFilter):org.apache.beam.sdk.extensions.sketching.BloomFilters$FilterByBloomFilterFn:filtered-out: 123456789))

Now they look like:

{\n \"metrics\": {\n \"attempted\": [{\n \"urn\": \"beam:metric:user:sum_int64:v1\",\n \"type\": \"beam:metrics:sum_int64:v1\",\n \"payload\": \"ip0K\",\n \"labels\": {\n \"NAMESPACE\": \"ImageClicksReader\",\n \"NAME\": \"filtered-out\",\n \"PTRANSFORM\": \"sets-by-key-bloomfilter/filter-by-bloom-filter/ParMultiDo(FilterByBloomFilter\"\n }\n } \n} \n}

 

Issue Links

• links to: GitHub Pull Request #14978