Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29072

Properly track shuffle write time with refactor

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Shuffle
    • Labels:
      None

      Description

      From SPARK-28209, SPARK-28570, and SPARK-28571, we used the new shuffle writer plugin API across all the shuffle writers. However, we accidentally lost time tracking metrics for shuffle writes in the process, particularly for UnsafeShuffleWriter when writing with streams (without transferTo), as well as the SortShuffleWriter.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mcheah Matt Cheah
                Reporter:
                mcheah Matt Cheah
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: