[BEAM-9065] Spark runner accumulates metrics (incorrectly) between runs

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.19.0
• Component/s: runner-spark
• Assignee: Etienne Chauchot
• Reporter: Etienne Chauchot
• Created: 08/Jan/20 10:28
• Updated: 13/Apr/23 10:59
• Resolved: 27/Jan/20 20:15

Description

When pipeline.run() is called, MetricsAccumulator (wrapper of MetricsContainerStepMap spark accumulator) is initialized. Spark needs this class to be a singleton for failover. The problem is that when several pipelines are run inside the same JVM, the initialization of MetricsAccumulator singleton does not reset the underlying spark accumulator  causing metrics to be accumulated between runs.

Issue Links

• fixes: BEAM-2409 Spark runner produces exactly twice the number of results in streaming mode when use triggers to re-window results on global window.
• fixes: BEAM-7156 Nexmark Query 14 'SESSION_SIDE_INPUT_JOIN' is producing twice the number of results in Spark Runner
• links to: GitHub Pull Request #10530
• links to: GitHub Pull Request #10670