Description
Kafka source (for SQL/SS) provides options to set specific offset per topic partition to let source starts reading from start offsets and ends reading until end offsets. ("startingOffsets" and "endingOffsets" in below document.)
http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
I'd like to introduce new options "startingOffsetsByTimestamp" and "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're unlikely to set the different value per partition) to let source starts reading from offsets which have equal of greater timestamp, and ends reading until offsets which have equal of greater timestamp.
The option would be optional of course, and have a preference to apply the options 1) timestamp option applies first, and if it doesn't exist, offset option applies.
Attachments
Issue Links
- is duplicated by
-
SPARK-26849 Introduce new option to Kafka source: offset by timestamp (starting/ending)
-
- Closed
-
- links to