Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4764

Ensure that files are fetched atomically

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.2, 1.2.1, 1.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      It does not seem necessary in the doFetchFile method to first download the file into a local directory then move it (it may cause a copy instead of a rename).

      tempFile (where the file is downloaded) should be created in the same directory than targetFile, so that the move from tempFile to targetFile is always atomic

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                preaudc Christophe Préaud
                Reporter:
                preaudc Christophe Préaud
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: