[SPARK-20776] Fix JobProgressListener perf. problems caused by empty TaskMetrics initialization

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 16/May/17 22:35
• Updated: 18/May/17 18:00
• Resolved: 17/May/17 05:05

Description

In

./bin/spark-shell --master=local[64]

I ran

  sc.parallelize(1 to 100000, 100000).count()

and profiled the time spend in the LiveListenerBus event processing thread. I discovered that the majority of the time was being spent constructing empty TaskMetrics instances inside JobProgressListener. As I'll show in a PR, we can slightly simplify the code to remove the need to construct one empty TaskMetrics per onTaskSubmitted event.

Issue Links

• links to: [Github] Pull Request #18008 (JoshRosen)