Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
-
None
Description
In Kinesis we can start reading the stream at some point in the past during the retention period (up to 7 days). With current approach for setting record's timestamp and watermark (both are always set to current time, i.e. Instant.now()), we can't observe the actual position in the stream.
So the idea is to change this behaviour and set the record timestamp based on the ApproximateArrivalTimestamp. Watermark will be set accordingly to the last read record's timestamp.
ApproximateArrivalTimestamp is still some approximation and may result in having records with out-of-order timestamp's which in turn may result in some events marked as late. This however should not be a frequent issue and even if it happens it should be a matter of milliseconds or seconds so can be handled even with a tiny allowedLateness setting
Attachments
Issue Links
- links to
