[SPARK-17304] TaskSetManager.abortIfCompletelyBlacklisted is a perf. hotspot in scheduler benchmark

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0
• Component/s: Scheduler, Spark Core
• Target Version/s: 2.1.0
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 29/Aug/16 23:37
• Updated: 17/May/20 17:46
• Resolved: 30/Aug/16 20:15

Description

If you run

sc.parallelize(1 to 100000, 100000).map(identity).count()

then TaskSetManager.abortIfCompletelyBlacklisted() is the number-one performance hotspot in the scheduler, accounting for over half of the time. This method was introduced in SPARK-15865, so this is a performance regression in 2.1.0-SNAPSHOT.

Issue Links

• is broken by: SPARK-15865 Blacklist should not result in job hanging with less than 4 executors
• links to: [Github] Pull Request #14871 (JoshRosen)