Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23153

Support application dependencies in submission client's local file system

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      Currently local dependencies are not supported with Spark on K8S i.e. if the user has code or dependencies only on the client where they run spark-submit then the current implementation has no way to make those visible to the Spark application running inside the K8S pods that get launched. This limits users to only running applications where the code and dependencies are either baked into the Docker images used or where those are available via some external and globally accessible file system e.g. HDFS which are not viable options for many users and environments

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                liyinan926 Yinan Li
              • Votes:
                2 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated: