Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-3596

Kafka Streams: Window expiration needs to consider more than event time

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      Currently in Kafka Streams, the way the windows are expired in RocksDB is triggered by new event insertion. When a window is created at T0 with 10 minutes retention, when we saw a new record coming with event timestamp T0 + 10 +1, we will expire that window (remove it) out of RocksDB.

      In the real world, it's very easy to see event coming with future timestamp (or out-of-order events coming with big time gaps between events), this way of retiring a window based on one event's event timestamp is dangerous. I think at least we need to consider both the event's event time and server/stream time elapse.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hcai@pinterest.com Henry Cai
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: