[SPARK-4764] Ensure that files are fetched atomically

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.2, 1.2.1, 1.3.0
• Component/s: Spark Core
• Target Version/s: 1.1.2, 1.2.1, 1.3.0
• Assignee: Christophe Préaud
• Reporter: Christophe Préaud
• Created: 05/Dec/14 14:22
• Updated: 17/Dec/14 20:11
• Resolved: 17/Dec/14 20:11

Description

It does not seem necessary in the doFetchFile method to first download the file into a local directory then move it (it may cause a copy instead of a rename).

tempFile (where the file is downloaded) should be created in the same directory than targetFile, so that the move from tempFile to targetFile is always atomic

Issue Links

• is related to: SPARK-3967 Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
• links to: [Github] Pull Request #2855 (preaudc)