Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4144

Allow per stream/table timestamp extractor

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.1
    • Fix Version/s: 0.11.0.0
    • Component/s: streams
    • Labels:

      Description

      At the moment the timestamp extractor is configured via a StreamConfig value to KafkaStreams. That means you can only have a single timestamp extractor per app, even though you may be joining multiple streams/tables that require different timestamp extraction methods.

      You should be able to specify a timestamp extractor via KStreamBuilder.stream()/table(), just like you can specify key and value serdes that override the StreamConfig defaults.

      KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788

      Specifying a per-stream extractor should only be possible for sources, but not for intermediate topics. For PAPI we cannot enforce this, but for DSL through() should not allow to set a custom extractor by the user. In contrast, with regard to KAFKA-4785, is must internally set an extractor that returns the record's metadata timestamp in order to overwrite the global extractor from StreamsConfig (ie, set FailOnInvalidTimestampExtractor). This change should be done in KAFKA-4785 though.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jeyhunkarimov Jeyhun Karimov
                Reporter:
                elevy Elias Levy
                Reviewer:
                Matthias J. Sax
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: