Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7442

Bounded Reads for Flink Runner fails with OOM

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.14.0
    • Component/s: runner-flink
    • Labels:
      None

      Description

      When Flink runner is reading from a bounded source and if the total number of files are huge and the count is more. FlinkRunner throws an OOM error. This is happening because the current implementation doesn't read them sequentially but simultaneously thus causing all of the files to be in memory which quickly breaks the cluster.

      Solution : To wrap `UnboundedReadFromBoundedSource` class by a wrapper to see that when the stream is a bounded source we make it read it sequentially using a queue.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aiyangar Akshay Iyangar
                Reporter:
                aiyangar Akshay Iyangar
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m