[CALCITE-3073] Support to consume from timestamp in KafkaAdapter

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: kafka-adapter
• Labels: pull-request-available
• Assignee: Matt Wang
• Reporter: Mingmin Xu
• Created: 16/May/19 11:55
• Updated: 14/Jun/19 02:27

Description

Currently the KafkaAdapter consumes data from default offset(latest/earliest/last_offset) and runs forever.

In other words, if the app runs at first time and user want to consume the past data, user must set the value of 'auto.offset.reset' parameter to earliest.

auto.offset.reset：What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

• earliest: automatically reset the offset to the earliest offset
• latest: automatically reset the offset to the latest offset
• none: throw exception to the consumer if no previous offset is found for the consumer's group
• anything else: throw exception to the consumer.

for example, suppose data in Kafka is retained for 7 days and you just want to read from the data of yesterday, if you could not control the start timestamp, you can only read from the earliest offset, it's very inefficient. If supporting to consume from special timestamp in KafkaAdapter will be a good idea for some cases.

Issue Links

• is blocked by: CALCITE-2913 Adapter for Apache Kafka
• is related to: CALCITE-3080 support user-specified row schema in KafkaAdapter
• links to: GitHub Pull Request #1253