[SPARK-5581] When writing sorted map output file, avoid open / close between each partition

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 2.1.0
• Component/s: Shuffle, Spark Core
• Epic Link: Tungsten Phase 2
• Assignee: Brian Cho
• Reporter: Sandy Ryza
• Created: 04/Feb/15 02:39
• Updated: 17/May/20 18:30
• Resolved: 25/Jul/16 02:37

Description

      // Bypassing merge-sort; get an iterator by partition and just write everything directly.
      for ((id, elements) <- this.partitionedIterator) {
        if (elements.hasNext) {
          val writer = blockManager.getDiskWriter(
            blockId, outputFile, ser, fileBufferSize, context.taskMetrics.shuffleWriteMetrics.get)
          for (elem <- elements) {
            writer.write(elem)
          }
          writer.commitAndClose()
          val segment = writer.fileSegment()
          lengths(id) = segment.length
        }
      }

Issue Links

• links to: [Github] Pull Request #13382 (dafrista)