Description
Allowing to attach callbacks in kafka streams, to be triggered when a window is expired,
would help in use cases when the final state of the window is required.
It would be also useful if together with that functionally the user would be able to control whether the callback would be triggered in addition to emitting the normal change log down the stream, or only triggering the callback when the window is expired. (for example in cases when only the final window state is required, and any updates to the window state during the window time interval are not important)
An example for use case could be left join with proper sql semantics:
A left join B currently would emit (a1,null) and (a1,b1) if b1 arrives within the defined join time window.
what I would like is to have ONLY ONE result:
(a1,null) if no b1 arrived during the defined join time window, OR ONLY (a1,b1) if b1 did arrived in the specified join time window.
One possible solution could be to use the current kafka streams left join with downstream processor which would put the results in a windowed Ktable.
The window size would be same as for the left join operation, however, only the final state of that window would be emitted down the stream once the time window is expired.
So if the left join produces (a1, null) and after X minutes no (a1, b1) was produced, eventually only (a1, null) would be emitted, on the other hand, if the left join produces (a1, null) and after X-t minutes (a1, b1) was produced by the left join operation => only (a1, b1) would be emitted eventually down the stream after X minutes.
Another use case is when the window state is written to another kafka topic which is then used to persist the window states into a db, However, many times only the final window state
is required, and functionality to get only the last window state would help in reducing load from the db, since only the final window state would be persisted to the db, instead of multiple db writes for each window state update.
Attachments
Issue Links
- is related to
-
KAFKA-7224 KIP-328: Add spill-to-disk for Suppression
-
- Open
-
-
-
- Resolved
-
-
-
- Closed
-
- relates to
-
-
- Open
-