Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7413

Time to write shuffle spill files is not captured in ShuffleWriteMetrics

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • None
    • None
    • Shuffle, Spark Core

    Description

      In ExternalSorter's spillToMergeableFile() method, we pass ShuffleWriteMetrics instances to the disk writers, but discard the shuffleWriteTime metrics captured here. I think that we should account for this IO time, possibly by introducing new metrics to distinguish time spent writing spills vs. writing final shuffle output and extending the UI to break down the overall IO write time in terms of these two components.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-3577 Add task metric to report spill time

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is part of

          Umbrella - An overarching type made of sub-tasks SPARK-7416 Shuffle performance metrics umbrella

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              joshrosen Josh Rosen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: