Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28074

[SS] Log warn message on possible correctness issue for multiple stateful operations in single query

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Please refer https://lists.apache.org/thread.html/cc6489a19316e7382661d305fabd8c21915e5faf6a928b4869ac2b4a@%3Cdev.spark.apache.org%3E to see rationalization of the issue.

      tl;dr. Using multiple stateful operators without fully understanding how global watermark affects the query brings correctness issue.

      While we may not want to fully block the operations given some users could leverage it with understanding of global watermark, it would be necessary for end users to warn on such breaking-correctness possibility, and provide alternative(s).

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              kabhwan Jungtaek Lim
              Reporter:
              kabhwan Jungtaek Lim

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment