[SPARK-17562] I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Not A Problem
• Affects Version/s: 2.0.0
• Component/s: Spark Core
• Labels: easyfix performance
• Flags: Important
• Assignee: Unassigned
• Reporter: Jianfei Wang
• Created: 16/Sep/16 12:06
• Updated: 21/Sep/16 01:11
• Resolved: 21/Sep/16 01:11

Description

In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will never be executed, so we can remove them

else  {
writer.revertPartialWritersAndClose()
}

the source code is as below:

try {
      while (inMemoryIterator.hasNext) {
        val partitionId = inMemoryIterator.nextPartition()
        require(partitionId >= 0 && partitionId < numPartitions,
          s"partition Id: ${partitionId} should be in the range [0, ${numPartitions})")
        inMemoryIterator.writeNext(writer)
        elementsPerPartition(partitionId) += 1
        objectsWritten += 1

        if (objectsWritten == serializerBatchSize) {
          flush()
        }
      }
      if (objectsWritten > 0) {
        flush()
      } else {
        writer.revertPartialWritesAndClose()
   }
      success = true
    } finally {
      if (success) {
        writer.close()
      } else {
        // This code path only happens if an exception was thrown above before we set success;
        // close our stuff and let the exception be thrown further
        writer.revertPartialWritesAndClose()
        if (file.exists()) {
          if (!file.delete()) {
            logWarning(s"Error deleting ${file}")
          }
        }
      }
    }