Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18124

Observed delay based event time watermarks

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: Structured Streaming
    • Labels:
      None
    • Target Version/s:

      Description

      Whenever we aggregate data by event time, we want to consider data is late and out-of-order in terms of its event time. Since we keep aggregate keyed by the time as state, the state will grow unbounded if we keep around all old aggregates in an attempt consider arbitrarily late data. Since the state is a store in-memory, we have to prevent building up of this unbounded state. Hence, we need a watermarking mechanism by which we will mark data that is older beyond a threshold as “too late”, and stop updating the aggregates with them. This would allow us to remove old aggregates that are never going to be updated, thus bounding the size of the state.

      Here is the design doc - https://docs.google.com/document/d/1z-Pazs5v4rA31azvmYhu4I5xwqaNQl6ZLIS03xhkfCQ/edit?usp=sharing

        Attachments

          Activity

            People

            • Assignee:
              marmbrus Michael Armbrust
              Reporter:
              tdas Tathagata Das
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: