Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3577

Add task metric to report spill time

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.1.0
    • 2.3.0
    • Shuffle, Spark Core
    • None

    Description

      The ExternalSorter passes its own ShuffleWriteMetrics into ExternalSorter. The write time recorded in those metrics is never used. We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).

      Attachments

        1. spill_size.jpg
          172 kB
          Gaoxiang Liu

        Issue Links

          Activity

            People

              sitalkedia@gmail.com Sital Kedia
              kayousterhout Kay Ousterhout
              Votes:
              4 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: