Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33447

Stage level scheduling, allow specifying other spark configs via ResourceProfile

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • Spark Core
    • None

    Description

      With the addition of stage level scheduling and ResourceProfiles, we only allow certain resources to specified currently.  There are many other configs that users may want to change between changes. We should look at adding in perhaps SparkConf settings.

      One very specific one that was brought up in review was ability to change yarn queue (if that is even possible in yarn) between stages because someone might want ot use one queue for ETL and a separate queue for ML where there are nodes with GPU.  Or perhaps node labels separately.

      Attachments

        Issue Links

          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. SPARK-27495 SPIP: Support Stage level resource configuration and scheduling

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: