[SPARK-6313] Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1, 1.3.0
• Fix Version/s: 1.2.2, 1.3.1, 1.4.0
• Component/s: Spark Core
• Target Version/s: 1.3.1
• Assignee: Nathan McCarthy
• Reporter: Nathan McCarthy
• Created: 13/Mar/15 04:17
• Updated: 17/Mar/15 16:35
• Resolved: 17/Mar/15 16:34

Description

When running in cluster mode and mounting the spark work dir on a NFS volume (or some volume which doesn't support file locking), the fetchFile (used for downloading JARs etc on the executors) method in Spark Utils class will fail. This file locking was introduced as an improvement with SPARK-2713.

See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L415

Introduced in 1.2 in commit; https://github.com/apache/spark/commit/7aacb7bfad4ec73fd8f18555c72ef696

As this locking is for optimisation for fetching files, could we take a different approach here to create a temp/advisory lock file?

Typically you would just mount local disks (in say ext4 format) and provide this as a comma separated list however we are trying to run Spark on MapR. With MapR we can do a loop back mount to a volume on the local node and take advantage of MapRs disk pools. This also means we dont need specific mounts for Spark and improves the generic nature of the cluster.

Issue Links

• links to: [Github] Pull Request #5036 (nemccarthy)