[SPARK-36242] Ensure spill file closed before set success to true in ExternalSorter.spillMemoryIteratorToDisk method - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 3.0.3, 3.1.2, 3.2.0, 3.3.0
Fix Version/s: 3.2.0, 3.1.3, 3.0.4, 3.3.0
Component/s: Spark Core
Labels:
None

Description

The processes of ExternalSorter.spillMemoryIteratorToDisk and ExternalAppendOnlyMap.spillMemoryIteratorToDisk are similar, but there are some differences in setting `success = true`

Code of ExternalSorter.spillMemoryIteratorToDisk as follows:

      if (objectsWritten > 0) {
        flush()
      } else {
        writer.revertPartialWritesAndClose()
      }
      success = true
    } finally {
      if (success) {
        writer.close()
      } else {
        ...
      }
    }

Code of ExternalSorter.spillMemoryIteratorToDisk as follows:

  if (objectsWritten > 0) {
    flush()
    writer.close()
  } else {
    writer.revertPartialWritesAndClose()
  }
  success = true
} finally {
  if (!success) {
    ...
  }
}

It seems that the processing of `ExternalSorter.spillMemoryIteratorToDisk` mehod is more reasonable, We should make sure setting `success = true` after the spill file is closed