[SPARK-15703] Make ListenerBus event queue size configurable

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: Scheduler, Spark Core, Web UI
• Target Version/s: 2.0.1
• Assignee: Dhruve Ashar
• Reporter: Thomas Graves
• Created: 01/Jun/16 16:27
• Updated: 17/May/20 17:47
• Resolved: 26/Jul/16 18:25

Description

The Spark UI doesn't seem to be showing all the tasks and metrics.

I ran a job with 100000 tasks but Detail stage page says it completed 93029:

Summary Metrics for 93029 Completed Tasks

The Stages for all jobs pages list that only 89519/100000 tasks finished but its completed. The metrics for shuffled write and input are also incorrect.

I will attach screen shots.

I checked the logs and it does show that all the tasks actually finished.

16/06/01 16:15:42 INFO TaskSetManager: Finished task 59880.0 in stage 2.0 (TID 54038) in 265309 ms on 10.213.45.51 (100000/100000)
16/06/01 16:15:42 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool

Issue Links

• is duplicated by: SPARK-18550 Make the queue capacity of LiveListenerBus configurable.
• is related to: SPARK-21460 Spark dynamic allocation breaks when ListenerBus event queue runs full
• is related to: SPARK-20858 Document ListenerBus event queue size property
• relates to: SPARK-16441 Spark application hang when dynamic allocation is enabled
• links to: [Github] Pull Request #14269 (dhruve)
• links to: [Github] Pull Request #15222 (zsxwing)