Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23287

Spark scheduler does not remove initial executor if not one job submitted

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • 2.2.1
    • None
    • Mesos, Scheduler, Spark Core

    Description

      When spark application submitted it deploy initial number of executors. If none of job has been submitted to application spark doesn't remove initial executor.

       
      Cluster manager - Mesos 1.4.1

      Spark 2.2.1

      spark app configuration:

      spark.dynamicAllocation.minExecutors=0

      spark.dynamicAllocation.executorIdleTimeout=25s

      spark.dynamicAllocation.initialExecutors=1

      spark.dynamicAllocation.schedulerBacklogTimeout=4s

      spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=5s

      Attachments

        Activity

          People

            Unassigned Unassigned
            humanmashine Pavel Plotnikov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: