Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28764

Remove unnecessary writePartitionedFile method from ExternalSorter

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: Shuffle, Tests
    • Labels:
      None

      Description

      Following SPARK-28571, we now use ExternalSorter#writePartitionedData in SortShuffleWriter when persisting the shuffle data via the shuffle writer plugin. However, we left the writePartitionedFile method on ExternalSorter strictly for tests. We should figure out a way how to refactor those tests to use writePartitionedData instead of writePartitionedFile.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mcheah Matt Cheah
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: