Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26848

Introduce new option to Kafka source - specify timestamp to start and end offset

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Kafka source (for SQL/SS) provides options to set specific offset per topic partition to let source starts reading from start offsets and ends reading until end offsets. ("startingOffsets" and "endingOffsets" in below document.)

      http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html

      I'd like to introduce new options "startingOffsetsByTimestamp" and "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're unlikely to set the different value per partition) to let source starts reading from offsets which have equal of greater timestamp, and ends reading until offsets which have equal of greater timestamp.

      The option would be optional of course, and have a preference to apply the options 1) timestamp option applies first, and if it doesn't exist, offset option applies.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kabhwan Jungtaek Lim
                Reporter:
                kabhwan Jungtaek Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: