Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6313

Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.2.1, 1.3.0
    • Fix Version/s: 1.2.2, 1.3.1, 1.4.0
    • Component/s: Spark Core
    • Labels:
      None
    • Target Version/s:

      Description

      When running in cluster mode and mounting the spark work dir on a NFS volume (or some volume which doesn't support file locking), the fetchFile (used for downloading JARs etc on the executors) method in Spark Utils class will fail. This file locking was introduced as an improvement with SPARK-2713.

      See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L415

      Introduced in 1.2 in commit; https://github.com/apache/spark/commit/7aacb7bfad4ec73fd8f18555c72ef696

      As this locking is for optimisation for fetching files, could we take a different approach here to create a temp/advisory lock file?

      Typically you would just mount local disks (in say ext4 format) and provide this as a comma separated list however we are trying to run Spark on MapR. With MapR we can do a loop back mount to a volume on the local node and take advantage of MapRs disk pools. This also means we dont need specific mounts for Spark and improves the generic nature of the cluster.

        Attachments

          Activity

            People

            • Assignee:
              nemccarthy Nathan McCarthy
              Reporter:
              nemccarthy Nathan McCarthy
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: