Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8962

FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.19.0
    • Component/s: runner-flink
    • Labels:
      None

      Description

      The FlinkMetricContainer wraps the Beam metric container for reporting metrics, but also stores them as Flink accumulators. With high parallelism jobs with over a thousand tasks and many built-in Beam metrics for every Beam step, this can accumulate to over 100MB of serialized data which is stored in the JobManager's ExecutionGraph. This then fails to even sent over the wire, due to the akka.framesize limit (10MB by default), and manifests in {{500 Internal Server Error}}s in the web frontend.

      We need to introduce an option to disable the reporting via accumulators. It is mostly useful for batch workloads where you can retrieve the final accumulator values at the end of the job. It adds a lot of memory and network overhead.

      Perhaps we could even turn off the accumulators for streaming jobs, or entirely and make them opt-in.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mxm Maximilian Michels
                Reporter:
                mxm Maximilian Michels
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 50m
                  2h 50m