Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27053

How about allowing engineers to use a different ExecutorBackend in StandAlone mode?

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Invalid
    • Affects Version/s: 2.3.3
    • Fix Version/s: None
    • Component/s: Deploy, Spark Submit
    • Labels:
      None

      Description

      In Standalone mode, the command for starting an Executor JVM in is hardcoded to use org.apache.spark.executor.CoarseGrainedExecutorBackend. There seems to be no way to configure the submit operation to use a custom ExecutorBackend (a subclass of CoarseGrainedExecutorBackend).

      This is very useful when engineers need to initialize things like starting a JDBC connection and Closing JDBC connection once per Executor.

      At line 103 of StandaloneSchedulerBackend.scala, why not make the fully qualified name of the executor backend class configurable? And then fall back to this default executor backend class if it's not configured.

      val command = Command("org.apache.spark.executor.CoarseGrainedExecutorBackend",
            args, sc.executorEnvs, classPathEntries ++ testingClassPath, libraryPathEntries, javaOpts)

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              briggxs Ross Brigoli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: