Details
Description
Kafka does not support negative record timestamps, and this prevents the storage of historical data in Kafka. In general, negative timestamps are supported by UNIX system time stamps:
From https://en.wikipedia.org/wiki/Unix_time
The Unix time number is zero at the Unix epoch, and increases by exactly 86,400 per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after the epoch, is represented by the Unix time number 12,677 × 86,400 = 1095292800. This can be extended backwards from the epoch too, using negative numbers; thus 1957-10-04T00:00:00Z, 4,472 days before the epoch, is represented by the Unix time number −4,472 × 86,400 = −386380800.
Allowing for negative timestamps would require multiple changes:
- while brokers in general do support negative timestamps, broker use -1 as default value if a producer uses an old message format (this would not be compatible with supporting negative timestamps "end-to-end" as -1 cannot be used as "unknown" anymore): we could introduce a message flag indicating a missing timestamp (and let producer throw an exception if ConsumerRecord#timestamp() is called. Another possible solution might be, to require topics that are used by old producers to be configured with LogAppendTime semantics and rejecting writes to topics with CreateTime semantics for older message formats
- KafkaProducer does not allow to send records with negative timestamp and thus this would need to be fixed
- Streams API does drop records with negative timestamps (or fails by default) – also, some internal store implementation for windowed stores assume that there are not negative timestamps to do range queries
There might be other gaps we need to address. This is just a summary of issue coming to my mind atm.
Attachments
Issue Links
- is related to
-
KAFKA-6169 Kafka Log should reject negative timestamps
-
- Resolved
-
- relates to
-
KAFKA-4393 Improve invalid/negative TS handling
-
- Resolved
-