Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19617

Fix the race condition when starting and stopping a query quickly

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.2, 2.1.0
    • 2.1.1, 2.2.0
    • Structured Streaming
    • None

    Description

      The streaming thread in StreamExecution uses the following ways to check if it should exit:

      • Catch an InterruptException.
      • `StreamExecution.state` is TERMINATED.

      when starting and stopping a query quickly, the above two checks may both fail.

      • Hit HADOOP-14084 and swallow InterruptException
      • StreamExecution.stop is called before `state` becomes `ACTIVE`. Then runBatches changes the state from `TERMINATED` to `ACTIVE`.

      If the above cases both happen, the query will hang forever.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            zsxwing Shixiong Zhu
            zsxwing Shixiong Zhu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment