[SPARK-18838] High latency of event processing for large jobs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.3.0
• Assignee: Marcelo Masiero Vanzin
• Reporter: Sital Kedia
• Created: 13/Dec/16 04:00
• Updated: 08/Feb/18 14:52
• Resolved: 20/Sep/17 05:42

Description

Currently we are observing the issue of very high event processing delay in driver's `ListenerBus` for large jobs with many tasks. Many critical component of the scheduler like `ExecutorAllocationManager`, `HeartbeatReceiver` depend on the `ListenerBus` events and this delay might hurt the job performance significantly or even fail the job. For example, a significant delay in receiving the `SparkListenerTaskStart` might cause `ExecutorAllocationManager` manager to mistakenly remove an executor which is not idle.

The problem is that the event processor in `ListenerBus` is a single thread which loops through all the Listeners for each event and processes each event synchronously https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala#L94. This single threaded processor often becomes the bottleneck for large jobs. Also, if one of the Listener is very slow, all the listeners will pay the price of delay incurred by the slow listener. In addition to that a slow listener can cause events to be dropped from the event queue which might be fatal to the job.

To solve the above problems, we propose to get rid of the event queue and the single threaded event processor. Instead each listener will have its own dedicate single threaded executor service . When ever an event is posted, it will be submitted to executor service of all the listeners. The Single threaded executor service will guarantee in order processing of the events per listener. The queue used for the executor service will be bounded to guarantee we do not grow the memory indefinitely. The downside of this approach is separate event queue per listener will increase the driver memory footprint.

Issue Links

• is duplicated by: SPARK-21460 Spark dynamic allocation breaks when ListenerBus event queue runs full
• is duplicated by: SPARK-16441 Spark application hang when dynamic allocation is enabled
• is duplicated by: SPARK-18881 Spark never finishes jobs and stages, JobProgressListener fails
• is related to: SPARK-20863 Add metrics/instrumentation to LiveListenerBus
• is related to: SPARK-18975 Add an API to remove SparkListener from SparkContext
• relates to: SPARK-21460 Spark dynamic allocation breaks when ListenerBus event queue runs full
• relates to: SPARK-21303 Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill
• relates to: SPARK-18975 Add an API to remove SparkListener from SparkContext
• links to: [Github] Pull Request #16291 (sitalkedia)
• links to: [Github] Pull Request #18004 (bOOm-X)
• links to: [Github] Pull Request #18253 (bOOm-X)
• links to: [Github] Pull Request #19211 (vanzin)
• links to: [Github] Pull Request #19297 (vanzin)