Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3577

Add task metric to report spill time

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.1.0
    • 2.3.0
    • Shuffle, Spark Core
    • None

    Description

      The ExternalSorter passes its own ShuffleWriteMetrics into ExternalSorter. The write time recorded in those metrics is never used. We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).

      Attachments

        1. spill_size.jpg
          172 kB
          Gaoxiang Liu

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-7413 Time to write shuffle spill files is not captured in ShuffleWriteMetrics

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3172 Distinguish between shuffle spill on the map and reduce side

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #2504 [Github] Pull Request #2504 (sryza)

          Pull request #17471 [Github] Pull Request #17471 (sitalkedia)

          Activity

            People

              sitalkedia@gmail.com Sital Kedia
              kayousterhout Kay Ousterhout
              Votes:
              4 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: