Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9065

Spark runner accumulates metrics (incorrectly) between runs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.19.0
    • Component/s: runner-spark
    • Labels:
      None

      Description

      When pipeline.run() is called, MetricsAccumulator (wrapper of MetricsContainerStepMap spark accumulator) is initialized. Spark needs this class to be a singleton for failover. The problem is that when several pipelines are run inside the same JVM, the initialization of MetricsAccumulator singleton does not reset the underlying spark accumulator  causing metrics to be accumulated between runs.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                echauchot Etienne Chauchot
                Reporter:
                echauchot Etienne Chauchot
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h