Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9733

ImpulseSourceFunction does not emit a final watermark

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P1
    • Resolution: Fixed
    • None
    • 2.21.0
    • runner-flink
    • None

    Description

      The Flink Runner's ImpulseSourceFunction does not emit a final watermark, unless --shutdownSourcesOnFinalWatermark flag has been specified (the flag is used in tests to shutdown the pipeline after reading all data). Most pipelines will be long-running and thus do not specify the flag.

      Not sending out the final watermark causes GroupByKey to hold back the data of event time windows until the pipeline is shut down (the final watermark is always emitted on pipeline shutdown which is why using the above flag works).

      Attachments

        Issue Links

          Activity

            People

              mxm Maximilian Michels
              mxm Maximilian Michels
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 11h
                  11h