Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8314

Beam Fn Api metrics piling causes pipeline to stuck after running for a while

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.16.0
    • Component/s: runner-dataflow
    • Labels:
      None

      Description

      Seems that in StreamingDataflowWorker we are not able to update the metrics fast enough to dataflow service, the piling metrics causes memory usage to increase and eventually leads to excessive memory thrashing/GC. And it will almost stop the pipeline from processing new items.

       

       

        Attachments

        1. E4UaSUhJJKF.png
          339 kB
          Yichi Zhang

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              yichi Yichi Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 20m
                5h 20m