[KAFKA-3596] Kafka Streams: Window expiration needs to consider more than event time

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 0.10.0.0
• Component/s: streams
• Labels: architecture
• Assignee: Unassigned
• Reporter: Henry Cai
• Created: 20/Apr/16 21:44
• Updated: 15/Jan/20 22:19
• Resolved: 15/Jan/20 22:19

Description

Currently in Kafka Streams, the way the windows are expired in RocksDB is triggered by new event insertion. When a window is created at T0 with 10 minutes retention, when we saw a new record coming with event timestamp T0 + 10 +1, we will expire that window (remove it) out of RocksDB.

In the real world, it's very easy to see event coming with future timestamp (or out-of-order events coming with big time gaps between events), this way of retiring a window based on one event's event timestamp is dangerous. I think at least we need to consider both the event's event time and server/stream time elapse.