Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20814

Mesos scheduler does not respect spark.executor.extraClassPath configuration

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.2.0
    • 2.2.0
    • Mesos
    • None

    Description

      When Spark executors are deployed on Mesos, the Mesos scheduler no longer respects the "spark.executor.extraClassPath" configuration parameter.

      MesosCoarseGrainedSchedulerBackend used to use the environment variable "SPARK_CLASSPATH" to add the value of "spark.executor.extraClassPath" to the executor classpath. However, "SPARK_CLASSPATH" was deprecated, and was removed in this commit https://github.com/apache/spark/commit/8f0490e22b4c7f1fdf381c70c5894d46b7f7e6fb#diff-387c5d0c916278495fc28420571adf9eL178.

      This effectively broke the ability for users to specify "spark.executor.extraClassPath" for Spark executors deployed on Mesos.

      Attachments

        Activity

          People

            vanzin Marcelo Masiero Vanzin
            gpang Gene Pang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: