[BEAM-10294] Beam metrics are unreadable in Spark history server

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.13.0
• Fix Version/s: 2.24.0
• Component/s: runner-spark
• Assignee: David Janicek
• Reporter: David Janicek
• Created: 22/Jun/20 11:30
• Updated: 23/Mar/21 16:58
• Resolved: 23/Mar/21 16:58

Description

Beam metrics shown in Spark history server are not readable. They're rendered as JSON which is created from protobuffer defined in beam_job_api.proto where metric's value is defined as bytes.

Similar issue was already addressed and fixed in BEAM-6062 but was broken by BEAM-4552.

Solution could be using SparkMetricsContainerStepMap instead of MetricsContainerStepMap inside MetricsContainerStepMapAccumulator as in BEAM-6062.

 

 

Issue Links

• relates to: BEAM-10719 Show metrics in a (more) human-readable format
• links to: GitHub Pull Request #12063