- Users use Apache Livy to submit spark jobs. Livy pod's spark-submit command will be invoked to submit jobs to k8e cluster with --deploy-mode cluster
- The Livy pod (submission client) does NOT have permission (secure token) to access remote dependencies specified by --files, --jars, --py-files from S3, gcs, hdfs:// . Only future driver pod and executor pod will have permissions with dynamic secure tokens to download those remote files.
Is it possible to support options to disable remote resource resolution/download at Livy pod or submission client host?
Of course, users/developers will make sure that
- Remote files are accessible by driver and executor pods with secure tokens.
- files with local paths are placed into driver and executor in advance.