Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18721

ForeachSink breaks Watermark in append mode

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.1.0
    • 2.1.0
    • Structured Streaming
    • None

    Description

      The watermark is not updated in append mode with a ForeachSink

      Because ForeachSink creates a separate IncrementalExecution instance, the physical plan will be recreated for the logical plan, which results in a new EventTimeWatermarkExec operator being created, that's unreachable from StreamExecution. This results in the watermark never being updated, and append mode never emits results.

      Attachments

        Issue Links

          Activity

            People

              zsxwing Shixiong Zhu
              copris Cristian Opris
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: