Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15815

Hang while enable blacklistExecutor and DynamicExecutorAllocator

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 1.6.1
    • None
    • Scheduler, Spark Core
    • None

    Description

      Enable BlacklistExecutor with some time large than 120s and enabled DynamicAllocate with minExecutors = 0
      1. Assume there only left 1 task running in Executor A, and other Executor are all timeout.
      2. the task failed, so task will not scheduled in current Executor A due to enable blacklistTime.
      3. For ExecutorAllocateManager, it always request targetNumExecutor=1 executors, due to we already have executor A, so the oldTargetNumExecutor == targetNumExecutor = 1, so will never add more Executors...even if Executor A was timeout. it became endless request delta=0 executors.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            SuYan SuYan
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment