Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20672

Keep the `isStreaming` property in triggerLogicalPlan in Structured Streaming

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.1
    • None
    • Structured Streaming
    • None

    Description

      In Structured Streaming, the "isStreaming" property will be eliminated in each triggerLogicalPlan. Then, some rules will be applied to this triggerLogicalPlan mistakely. So, we should refactor existing code to better execute batch query and ss query.

      Attachments

        Issue Links

        Is contained by

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-21765 Ensure all leaf nodes that are derived from streaming sources have isStreaming=true

        • Major - Major loss of function.
        • Resolved
        is depended upon by

        Bug - A problem which impairs or prevents the functions of the product. SPARK-20373 Batch queries with 'Dataset/DataFrame.withWatermark()` does not execute

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. SPARK-19690 Join a streaming DataFrame with a batch DataFrame may not work

        • Critical - Crashes, loss of data, severe memory leak.
        • Resolved
        links to

        Pull request #17913 [Github] Pull Request #17913 (uncleGen)

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            uncleGen Genmao Yu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment