Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22870

Dynamic allocation should allow 0 idle time

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 2.3.0
    • Component/s: Scheduler
    • Labels:
      None

      Description

      As discussed in SPARK-22765, with SPARK-21656, an executor will not idle out when there are pending tasks to run. When there is no task to run, an executor will die out after spark.dynamicAllocation.executorIdleTimeout, which is currently required to be greater than zero. However, for efficiency, a user should be able to specify that an executor can die out immediately w/o being required to be idle for at least 1s.

      This is to make 0 a valid value for spark.dynamicAllocation.executorIdleTimeout, and special handling such a case might be needed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                q79969786 Yuming Wang
                Reporter:
                xuefuz Xuefu Zhang
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: