Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41146

Assert the offset range for Kafka source in Trigger.AvailableNow

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • Structured Streaming
    • None

    Description

      We encountered the issue where the data source did not properly implement the offset with Trigger.AvailableNow, and the query ran with processing same data continuously without stopping.

      We would like to proactively avoid such case for most used data sources.

      Attachments

        Issue Links

          is a clone of

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-41145 Assert the offset range for file stream source in Trigger.AvailableNow

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: