[SPARK-3577] Add task metric to report spill time

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 2.3.0
• Component/s: Shuffle, Spark Core
• Assignee: Sital Kedia
• Reporter: Kay Ousterhout
• Created: 17/Sep/14 22:59
• Updated: 29/Jun/17 06:27
• Resolved: 29/Jun/17 06:26

Description

The ExternalSorter passes its own ShuffleWriteMetrics into ExternalSorter. The write time recorded in those metrics is never used. We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).

Issue Links

• is duplicated by: SPARK-7413 Time to write shuffle spill files is not captured in ShuffleWriteMetrics
• is related to: SPARK-3172 Distinguish between shuffle spill on the map and reduce side
• links to: [Github] Pull Request #2504 (sryza)
• links to: [Github] Pull Request #17471 (sitalkedia)