Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17644

The failed stage never resubmitted due to abort stage in another thread

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersStop watchingWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.0, 2.0.0
    • 2.0.1, 2.1.0
    • Scheduler, Spark Core
    • None

    Description

      there is a race condition when FetchFailed and resubmit failed stage:
      job1, job2 run in different threads, if job 1 failed 4 times due to fetchfailed and aborted, then job2 can not post ResubmitFailedStages becase the failedStages in DAGScheduler is not empty now.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            scwf Fei Wang Assign to me
            scwf Fei Wang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment