[KAFKA-7748] Add wall clock TimeDefinition for suppression of intermediate events

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.1.0
• Component/s: streams
• Labels: needs-kip
• Assignee: Unassigned
• Reporter: Jonathan Gordon
• Created: 17/Dec/18 22:02
• Updated: 15/Apr/21 09:11

Description

Currently, Kafka Streams offers the ability to suppress intermediate events based on either RecordTime or WindowEndTime, which are in turn defined by stream time:

Suppressed.untilTimeLimit(final Duration timeToWaitForMoreEvents, final BufferConfig bufferConfig)

It would be helpful to have another option that would allow suppression of intermediate events based on wall clock time. This would allow us to only produce a limited number of aggregates independent of their stream time (which in our case is event time).

For reference, here's the relevant KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables#KIP-328:AbilitytosuppressupdatesforKTables-Best-effortratelimitperkey

And here's the relevant Confluent Slack thread:

https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1544468349201700