[SPARK-23626] DAGScheduler blocked due to JobSubmitted event

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.1, 2.3.3, 2.4.3, 3.0.0
• Fix Version/s: 3.1.3, 3.0.4, 3.2.1, 3.3.0
• Component/s: Scheduler, Spark Core
• Assignee: Josh Rosen
• Reporter: Ajith S
• Created: 08/Mar/18 06:28
• Updated: 14/Oct/21 21:44
• Resolved: 14/Oct/21 21:44

Description

DAGScheduler becomes a bottleneck in cluster when multiple JobSubmitted events has to be processed as DAGSchedulerEventProcessLoop is single threaded and it will block other tasks in queue like TaskCompletion.

The JobSubmitted event is time consuming depending on the nature of the job (Example: calculating parent stage dependencies, shuffle dependencies, partitions) and thus it blocks all the events to be processed.

 

I see multiple JIRA referring to this behavior

https://issues.apache.org/jira/browse/SPARK-2647

https://issues.apache.org/jira/browse/SPARK-4961

 

Similarly in my cluster some jobs partition calculation is time consuming (Similar to stack at SPARK-2647) hence it slows down the spark DAGSchedulerEventProcessLoop which results in user jobs to slowdown, even if its tasks are finished within seconds, as TaskCompletion Events are processed at a slower rate due to blockage.

Issue Links

• relates to: SPARK-2647 DAGScheduler plugs others when processing one JobSubmitted event
• relates to: SPARK-4961 Put HadoopRDD.getPartitions forward to reduce DAGScheduler.JobSubmitted processing time
• links to: [Github] Pull Request #20770 (AjithShetty2489)
• links to: [Github] Pull Request #34265 (JoshRosen)
• links to: GitHub Pull Request #20770
• links to: GitHub Pull Request #24438
• links to: GitHub Pull Request #27234