Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33083

Optionally skip remote/local dependency resolution at submission client in k8s cluster mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.4.7
    • Fix Version/s: None
    • Component/s: Kubernetes, Spark Submit
    • Labels:
      None

      Description

      Usecase:

      1. Users use Apache Livy to submit spark jobs.  Livy pod's spark-submit command will be invoked to submit jobs to k8e cluster with --deploy-mode cluster
      2. The Livy pod (submission client) does NOT have permission (secure token) to access remote dependencies specified by --files, --jars, --py-files from S3, gcs, hdfs:// . Only future driver pod and executor pod will have permissions with dynamic secure tokens to download those remote files.

      Is it possible to support options to disable remote resource resolution/download at Livy pod or submission client host? 

      Of course, users/developers will make sure that

      1. Remote files are accessible by driver and executor pods with secure tokens.
      2. files with local paths are placed into driver and executor in advance. 

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dexterhu Dexter Hu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 72h
                72h
                Remaining:
                Remaining Estimate - 72h
                72h
                Logged:
                Time Spent - Not Specified
                Not Specified