Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27963

Allow dynamic allocation without an external shuffle service

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:

      Description

      It would be useful for users to be able to enable dynamic allocation without the need to provision an external shuffle service. One immediate use case is the ability to use dynamic allocation on Kubernetes, which doesn't yet have that service.

      This has been suggested before (e.g. https://github.com/apache/spark/pull/24083, which was attached to the k8s-specific SPARK-24432), and can actually be done without affecting the internals of the Spark scheduler (aside from the dynamic allocation code).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: