Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7748

Add wall clock TimeDefinition for suppression of intermediate events

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      Currently, Kafka Streams offers the ability to suppress intermediate events based on either RecordTime or WindowEndTime, which are in turn defined by stream time:

      Suppressed.untilTimeLimit(final Duration timeToWaitForMoreEvents, final BufferConfig bufferConfig)

      It would be helpful to have another option that would allow suppression of intermediate events based on wall clock time. This would allow us to only produce a limited number of aggregates independent of their stream time (which in our case is event time).

      For reference, here's the relevant KIP:

      https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables#KIP-328:AbilitytosuppressupdatesforKTables-Best-effortratelimitperkey

      And here's the relevant Confluent Slack thread:

      https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1544468349201700

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jonathanpdx Jonathan Gordon
            • Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated: