[SPARK-1924] Make local:/ scheme work in more deploy modes

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 1.0.0
• Component/s: Deploy
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Xiangrui Meng
• Created: 25/May/14 21:04
• Updated: 21/May/19 04:15
• Resolved: 21/May/19 04:15

Description

A resource marked "local:/" is assumed to be available on the local file system of every node. In such case, no data copying over network should happen. I tested different deploy modes in v1.0, right now we only support local:/ scheme for the app jar and secondary jars in the following modes:

1) local (jars are copied to the working directory)
2) standalone client
3) yarn client

It doesn’t work for -files and python apps (-py-files and app.py). For the next release, we could support more deploy modes.

Issue Links

• is related to: SPARK-6355 Spark standalone cluster does not support local:/ url for jar file