Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8314

Beam Fn Api metrics piling causes pipeline to stuck after running for a while

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P0
    • Resolution: Fixed
    • None
    • 2.16.0
    • runner-dataflow
    • None

    Description

      Seems that in StreamingDataflowWorker we are not able to update the metrics fast enough to dataflow service, the piling metrics causes memory usage to increase and eventually leads to excessive memory thrashing/GC. And it will almost stop the pipeline from processing new items.

       

       

      Attachments

        1. E4UaSUhJJKF.png
          339 kB
          Yichi Zhang

        Activity

          People

            Unassigned Unassigned
            yichi Yichi Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 5h 20m
                5h 20m