Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-4896

Don't redundantly copy executor dependencies in Utils.fetchFile

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.2, 1.2.1, 1.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      This JIRA is spun off from a comment by Ryan Williams on SPARK-3967, quoted here:

      I've been debugging this issue as well and I think I've found an issue in org.apache.spark.util.Utils that is contributing to / causing the problem:

      Files.move on line 390 is called even if targetFile exists and tempFile and targetFile are equal.

      The check on line 379 seems to imply the desire to skip a redundant overwrite if the file is already there and has the contents that it should have.

      Gating the Files.move call on a further if (!targetFile.exists) fixes the issue for me; attached is a patch of the change.

      In practice all of my executors that hit this code path are finding every dependency JAR to already exist and be exactly equal to what they need it to be, meaning they were all needlessly overwriting all of their dependency JARs, and now are all basically no-op-ing in Utils.fetchFile; I've not determined who/what is putting the JARs there, why the issue only crops up in yarn-cluster mode (or --master yarn --deploy-mode cluster), etc., but it seems like either way this patch is probably desirable.

      I'm spinning this off into its own JIRA so that we can track the merging of https://github.com/apache/spark/pull/2848 separately (since we have multiple PRs that contribute to fixing the original issue).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rdub Ryan Williams
                Reporter:
                joshrosen Josh Rosen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: