Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-3073

Support to consume from timestamp in KafkaAdapter

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Currently the KafkaAdapter consumes data from default offset(latest/earliest/last_offset) and runs forever.

      In other words, if the app runs at first time and user want to consume the past data, user must set the value of 'auto.offset.reset' parameter to earliest.

      auto.offset.reset:What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

      • earliest: automatically reset the offset to the earliest offset
      • latest: automatically reset the offset to the latest offset
      • none: throw exception to the consumer if no previous offset is found for the consumer's group
      • anything else: throw exception to the consumer.

      for example, suppose data in Kafka is retained for 7 days and you just want to read from the data of yesterday, if you could not control the start timestamp, you can only read from the earliest offset, it's very inefficient. If supporting to consume from special timestamp in KafkaAdapter will be a good idea for some cases.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              wangm92 Matt Wang
              Reporter:
              mingmxu Mingmin Xu

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 40m
                3h 40m

                  Issue deployment