Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5972

Don't allow shrinking merging windows

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 1.1.0, 1.2.0, 1.3.0
    • 1.2.1, 1.3.0
    • API / DataStream
    • None

    Description

      A misbehaving MergingWindowAssigner can cause a merge that results in a window that is smaller than the span of all the merged windows. This, in itself is not problematic. It becomes problematic when the end timestamp of a window that was not late before merging is now earlier than the watermark (the timestamp is smaller than the watermark).

      There are two choices:

      • immediately process the window
      • drop the window

      processing the window will lead to late data downstream.

      The current behaviour is to silently drop the window but that logic has a bug: we only remove the dropped window from the MergingWindowSet but we don't properly clean up state and timers that the window still (possibly) has. We should fix this bug in the process of resolving this issue.

      We should either just fix the bug and still silently drop windows or add a check and throw an exception when the end timestamp falls below the watermark.

      Attachments

        Issue Links

          Activity

            People

              aljoscha Aljoscha Krettek
              aljoscha Aljoscha Krettek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: