Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3577

Add task metric to report spill time

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 2.3.0
    • Component/s: Shuffle, Spark Core
    • Labels:
      None

      Description

      The ExternalSorter passes its own ShuffleWriteMetrics into ExternalSorter. The write time recorded in those metrics is never used. We should probably add task metrics to report this spill time, since for shuffles, this would have previously been reported as part of shuffle write time (with the original hash-based sorter).

        Attachments

        1. spill_size.jpg
          172 kB
          Gaoxiang Liu

          Issue Links

            Activity

              People

              • Assignee:
                sitalkedia@gmail.com Sital Kedia
                Reporter:
                kayousterhout Kay Ousterhout
              • Votes:
                4 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: