Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30626

Add SPARK_APPLICATION_ID into driver pod env

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Kubernetes, Spark Core
    • Labels:
      None

      Description

      This should be a minor improvement.

      The use case is we want to look up environment variables and create application folder and redirect driver logs to application folder.  Executors has it and we want to make a change to driver as well. 

       

      Limits:
       cpu: 1024m
       memory: 896Mi
       Requests:
       cpu: 1
       memory: 896Mi
      Environment:
       SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
       SPARK_LOCAL_DIRS: /var/data/spark-9c315655-aba4-47fb-821c-30268d02af7e
       SPARK_CONF_DIR: /opt/spark/conf

       

      https://github.com/apache/spark/blob/afe70b3b5321439318a456c7d19b7074171a286a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala#L73-L79

      We need SPARK_APPLICATION_ID inside the pod to organize logs 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                seedjeffwan Jiaxin Shan
                Reporter:
                seedjeffwan Jiaxin Shan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: