Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6614

kafka-streams to configure internal topics message.timestamp.type=CreateTime

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      After fixing KAFKA-4785 all internal topics using built-in RecordMetadataTimestampExtractor to read timestamps.

      Which doesn't seem to work correctly out of box with kafka brokers configured with log.message.timestamp.type=LogAppendTime when using custom message timestamp extractor.

      Example use-case windowed grouping + aggregation on late data:

      KTable<Windowed<Tuple>, Long> summaries = in
         .groupBy((key, value) -> ......)
         .windowedBy(TimeWindows.of(TimeUnit.HOURS.toMillis(1l)))
         .count();

      when processing late events:

      1. custom timestamp extractor will pick up timestamp in the past from message (let's say hour ago)
      2. re-partition topic during grouping phase will be written back to kafka using timestamp from (1)
      3. kafka broker will ignore provided timestamp in (2) to favor ingestion time
      4. streams lib will read re-partitioned topic back with RecordMetadataTimestampExtractor
      5. and will get ingestion timestamp (3), which usually close to "now"
      6. window start/end will be incorrectly set based on "now" instead of original timestamp from payload

      Understand there are ways to configure per-topic timestamp type in kafka brokers to solve this, but it will be really nice if kafka-streams library can take care of it itself.

      To follow "least-surprise" principle.  If library relies on timestamp.type for topic it manages it should enforce it.

      CC Guozhang Wang based on user group email discussion.

        Attachments

          Activity

            People

            • Assignee:
              meghanagupta Meghana Gupta
              Reporter:
              dvsekhvalnov Dmitry Vsekhvalnov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: