Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9065

Spark runner accumulates metrics (incorrectly) between runs

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.19.0
    • Component/s: runner-spark
    • Labels:
      None

      Description

      When pipeline.run() is called, MetricsAccumulator (wrapper of MetricsContainerStepMap spark accumulator) is initialized. Spark needs this class to be a singleton for failover. The problem is that when several pipelines are run inside the same JVM, the initialization of MetricsAccumulator singleton does not reset the underlying spark accumulator  causing metrics to be accumulated between runs.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              echauchot Etienne Chauchot
              Reporter:
              echauchot Etienne Chauchot

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 2.5h
                2.5h

                  Issue deployment