Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36057 SPIP: Support Customized Kubernetes Schedulers
  3. SPARK-36060

Support backing off dynamic allocation increases if resources are "stuck"

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.2.0
    • None
    • Kubernetes
    • None

    Description

      In a over-subscribed environment we may enter a situation where our requests for more pods are not going to be fulfilled. Adding more requests for more pods is not going to help and may slow down the scheduler. We should detect this situation and hold off on increasing pod requests until the scheduler allocates more pods to us. We have a limited version of this in the Kube scheduler it's self but it would be better to plumb this all the way through to the DA logic.

      Attachments

        Issue Links

          is superceded by

          Sub-task - The sub-task of the issue SPARK-38455 Support driver/executor PodGroup templates

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-36052 Introduce pending pod limit for Spark on K8s

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              holden Holden Karau
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: