Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23133

Spark options are not passed to the Executor in Docker context

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: Kubernetes
    • Labels:
      None
    • Environment:

      Running Spark on K8s using supplied Docker image.

      Description

      Reproduce:

      1. Build image with `bin/docker-image-tool.sh`.
      2. Submit application to k8s. Set executor options, e.g. ` --conf "spark.executor. extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf"`
      3. Visit Spark UI on executor and notice that option is not set.

      Expected behavior: options from spark-submit should be correctly passed to executor.

      Cause:

      `SPARK_EXECUTOR_JAVA_OPTS` is not defined in `entrypoint.sh`

      https://github.com/apache/spark/blob/branch-2.3/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh#L70

      https://github.com/apache/spark/blob/branch-2.3/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh#L44-L45

        Attachments

          Activity

            People

            • Assignee:
              akorzhuev Andrew Korzhuev
              Reporter:
              akorzhuev Andrew Korzhuev
              Shepherd:
              Marcelo Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified