Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4393

Improve invalid/negative TS handling

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.2.0
    • Component/s: streams
    • Labels:

      Description

      Currently, Kafka Streams does not handle invalid/negative timestamps returned from the TimestampExtractor gracefully, but fails with an exception, because negative timestamps cannot get handled in a meaningful way for any time based (ie, window) operators like window aggregates and joins.

      We want to change Streams to a auto-drop behavior for negative timestamps for those records (without any further user notification about dropped record) to enable users to "step over" those records and keep going (instead of an exception). To guard the user from silently dropping messages by default (and kept current fail-fast behavior), we change the default extractor ConsumerRecordTimestampExtractor to check the extracted meta-data record timestamp and raise an exception if it is negative. Furthermore, we add a "drop-and-log" extractor, as this seems to be a common behavior user might want to have. For any other behavior, users can still provide a custom TS-Extractor implementation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mjsax Matthias J. Sax
                Reporter:
                mjsax Matthias J. Sax
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: