Kafka
  1. Kafka
  2. KAFKA-3846

Connect record types should include timestamps

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0.0
    • Fix Version/s: 0.10.1.0
    • Component/s: KafkaConnect
    • Labels:

      Description

      Timestamps were added to records in the previous release, however this does not get propagated automatically to Connect because it uses custom wrappers to add fields and rename some for clarity.

      The addition of timestamps should be trivial, but can be really useful (e.g. in sink connectors that would like to include timestamp info if available but when it is not stored in the value).

      This is public API so it will need a KIP despite being very uncontentious.

        Issue Links

          Activity

          Hide
          ASF GitHub Bot added a comment -

          GitHub user shikhar opened a pull request:

          https://github.com/apache/kafka/pull/1537

          KAFKA-3846: include timestamp in Connect record types

          KIP to come

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/shikhar/kafka kafka-3846

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/kafka/pull/1537.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1537


          commit 2dc7e91c1ec06fcf987dda07a63acf45e1b2d13e
          Author: Shikhar Bhushan <shikhar@confluent.io>
          Date: 2016-06-22T00:42:48Z

          KAFKA-3846: include timestamp in Connect record types; add Builder for `SourceRecord`

          `SinkRecord` gets `timestampType` and `timestamp`
          `SourceRecord` gets `timestamp`
          `SourceRecord.Builder` is the new preferred way to construct `SourceRecord`s


          Show
          ASF GitHub Bot added a comment - GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1537 KAFKA-3846 : include timestamp in Connect record types KIP to come You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka kafka-3846 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1537.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1537 commit 2dc7e91c1ec06fcf987dda07a63acf45e1b2d13e Author: Shikhar Bhushan <shikhar@confluent.io> Date: 2016-06-22T00:42:48Z KAFKA-3846 : include timestamp in Connect record types; add Builder for `SourceRecord` `SinkRecord` gets `timestampType` and `timestamp` `SourceRecord` gets `timestamp` `SourceRecord.Builder` is the new preferred way to construct `SourceRecord`s
          Show
          Shikhar Bhushan added a comment - https://cwiki.apache.org/confluence/display/KAFKA/KIP-65%3A+Expose+timestamps+to+Connect
          Hide
          Ewen Cheslack-Postava added a comment -

          Issue resolved by pull request 1537
          https://github.com/apache/kafka/pull/1537

          Show
          Ewen Cheslack-Postava added a comment - Issue resolved by pull request 1537 https://github.com/apache/kafka/pull/1537
          Hide
          ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/kafka/pull/1537

          Show
          ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1537
          Hide
          Michael Andre Pearce (IG) added a comment -

          This looks great, is any documents updated to show how we configure bits like TimebasedPartitioner mentioned here http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/configuration_options.html to use this?

          Also is it possible to store this timestamp ( and other meta) in the hdfs/hive record it seems we can only store the payload. From the config options in connect.

          Show
          Michael Andre Pearce (IG) added a comment - This looks great, is any documents updated to show how we configure bits like TimebasedPartitioner mentioned here http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/configuration_options.html to use this? Also is it possible to store this timestamp ( and other meta) in the hdfs/hive record it seems we can only store the payload. From the config options in connect.
          Hide
          Ewen Cheslack-Postava added a comment -

          Michael Andre Pearce (IG) Since the timestamp wasn't available until recently, the existing connector has to use wall-clock time. I imagine it'd be a simple patch to use the timestamp, if available, from the record instead.

          Show
          Ewen Cheslack-Postava added a comment - Michael Andre Pearce (IG) Since the timestamp wasn't available until recently, the existing connector has to use wall-clock time. I imagine it'd be a simple patch to use the timestamp, if available, from the record instead.

            People

            • Assignee:
              Shikhar Bhushan
              Reporter:
              Ewen Cheslack-Postava
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development