[SPARK-33083] Optionally skip remote/local dependency resolution at submission client in k8s cluster mode

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.4.7
• Component/s: Kubernetes, Spark Submit
• Assignee: Unassigned
• Reporter: Dexter Hu
• Created: 07/Oct/20 07:43
• Updated: 07/Oct/20 08:08

Description

Usecase:

1. Users use Apache Livy to submit spark jobs.  Livy pod's spark-submit command will be invoked to submit jobs to k8e cluster with --deploy-mode cluster
2. The Livy pod (submission client) does NOT have permission (secure token) to access remote dependencies specified by --files, --jars, --py-files from S3, gcs, hdfs:// . Only future driver pod and executor pod will have permissions with dynamic secure tokens to download those remote files.

Is it possible to support options to disable remote resource resolution/download at Livy pod or submission client host? 

Of course, users/developers will make sure that

1. Remote files are accessible by driver and executor pods with secure tokens.
2. files with local paths are placed into driver and executor in advance.