Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6048

Support negative record timestamps

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: clients, core, streams
    • Labels:

      Description

      Kafka does not support negative record timestamps, and this prevents the storage of historical data in Kafka. In general, negative timestamps are supported by UNIX system time stamps:

      From https://en.wikipedia.org/wiki/Unix_time

      The Unix time number is zero at the Unix epoch, and increases by exactly 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after the epoch, is represented by the Unix time number 12,677 × 86,400 = 1095292800. This can be extended backwards from the epoch too, using negative numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is represented by the Unix time number −4,472 × 86,400 = −386380800.

      Allowing for negative timestamps would require multiple changes:

      • while brokers in general do support negative timestamps, broker use -1 as default value if a producer uses an old message format (this would not be compatible with supporting negative timestamps "end-to-end" as -1 cannot be used as "unknown" anymore): we could introduce a message flag indicating a missing timestamp (and let producer throw an exception if ConsumerRecord#timestamp() is called. Another possible solution might be, to require topics that are used by old producers to be configured with LogAppendTime semantics and rejecting writes to topics with CreateTime semantics for older message formats
      • KafkaProducer does not allow to send records with negative timestamp and thus this would need to be fixed
      • Streams API does drop records with negative timestamps (or fails by default) – also, some internal store implementation for windowed stores assume that there are not negative timestamps to do range queries

      There might be other gaps we need to address. This is just a summary of issue coming to my mind atm.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                chuhlomin Konstantin Chukhlomin
                Reporter:
                mjsax Matthias J. Sax
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: