[SPARK-21460] Spark dynamic allocation breaks when ListenerBus event queue runs full

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.0.0, 2.0.2, 2.1.0, 2.1.1, 2.2.0
• Component/s: Scheduler, Spark Core, YARN
• Labels: dynamic_allocation performance scheduler yarn

Environment

Spark 2.1
Hadoop 2.6

• Assignee: Unassigned
• Reporter: Ruslan Dautkhanov
• Created: 18/Jul/17 17:02
• Updated: 17/May/20 17:46
• Resolved: 08/Feb/18 14:52

Description

When ListenerBus event queue runs full, spark dynamic allocation stops working - Spark fails to shrink number of executors when there are no active jobs (Spark driver "thinks" there are active jobs since it didn't capture when they finished) .

ps. What's worse it also makes Spark flood YARN RM with reservation requests, so YARN preemption doesn't function properly too (we're on Spark 2.1 / Hadoop 2.6).

Issue Links

• duplicates: SPARK-18838 High latency of event processing for large jobs
• is related to: SPARK-18838 High latency of event processing for large jobs
• relates to: SPARK-15703 Make ListenerBus event queue size configurable