[SPARK-15815] Hang while enable blacklistExecutor and DynamicExecutorAllocator

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.6.1
• Component/s: Scheduler, Spark Core
• Assignee: Unassigned
• Reporter: SuYan
• Created: 08/Jun/16 06:23
• Updated: 06/Nov/18 14:25
• Resolved: 06/Nov/18 14:25

Description

Enable BlacklistExecutor with some time large than 120s and enabled DynamicAllocate with minExecutors = 0
1. Assume there only left 1 task running in Executor A, and other Executor are all timeout.
2. the task failed, so task will not scheduled in current Executor A due to enable blacklistTime.
3. For ExecutorAllocateManager, it always request targetNumExecutor=1 executors, due to we already have executor A, so the oldTargetNumExecutor == targetNumExecutor = 1, so will never add more Executors...even if Executor A was timeout. it became endless request delta=0 executors.

Issue Links

• Blocked: SPARK-8426 Add blacklist mechanism for YARN container allocation
• is duplicated by: SPARK-22148 TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled
• is duplicated by: SPARK-21539 Job should not be aborted when dynamic allocation is enabled or spark.executor.instances larger then current allocated number by yarn
• is related to: SPARK-22148 TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled
• relates to: SPARK-16554 Spark should kill executors when they are blacklisted
• relates to: SPARK-24413 Executor Blacklisting shouldn't immediately fail the application if dynamic allocation is enabled and no active executors
• relates to: SPARK-21539 Job should not be aborted when dynamic allocation is enabled or spark.executor.instances larger then current allocated number by yarn
• links to: [Github] Pull Request #14765 (suyanNone)
• links to: [Github] Pull Request #22288 (dhruve)