Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45178

Fallback to use single batch executor for Trigger.AvailableNow with unsupported sources rather than using wrapper

    XMLWordPrintableJSON

Details

    Description

      We have observed the case where wrapper implementation of Trigger.AvailableNow (
      AvailableNowDataStreamWrapper and subclasses) is not fully compatible with 3rd party data source and brought up correctness issue.
       
      While we could persuade 3rd party data source to support Trigger.AvailableNow, pursuing all 3rd parties to do this is too aggressive and challenging goal we never be able to make. Also, it may not be also possible to come up with the wrapper implementation which would have zero issue with any arbitrary source.
       
      As a mitigation, we want to make a slight behavioral change for such case, falling back to single batch execution (a.k.a. Trigger.Once) rather than using wrapper implementation. The exact behavior between Trigger.AvailableNow and Trigger.Once are different so it's technically behavioral change, but it's probably lot less surprised than failing the query.
       
      For extreme case where users are confident that there will be no issue at all on using wrapper, we will come up with a flag to provide the previous behavior.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: