Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.1.0
-
None
Description
The ExternalSorter passes its own ShuffleWriteMetrics into ExternalSorter. The write time recorded in those metrics is never used. We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).
Attachments
Attachments
Issue Links
- is duplicated by
-
SPARK-7413 Time to write shuffle spill files is not captured in ShuffleWriteMetrics
- Resolved
- is related to
-
SPARK-3172 Distinguish between shuffle spill on the map and reduce side
- Resolved
- links to