Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24793

Make spark-submit more useful with k8s

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.0
    • Fix Version/s: None
    • Component/s: Kubernetes
    • Labels:
      None

      Description

      Support controlling the lifecycle of Spark Application through spark-submit.
      For example:

      {{
      --kill app_name If given, kills the driver specified.
      --status app_name If given, requests the status of the driver specified.
      }}

      Potentially also --list to list all spark drivers running.

      Given that our submission client can actually launch jobs into many different namespaces, we'll need an additional specification of the namespace through a --namespace flag potentially.
      I think this is pretty useful to have instead of forcing a user to use kubectl to manage the lifecycle of any k8s Spark Application.

        Attachments

          Activity

            People

            • Assignee:
              foxish Anirudh Ramanathan
              Reporter:
              foxish Anirudh Ramanathan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: