Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698

[Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: capacity scheduler
    • Labels:

      Description

      We see some users want to migrate from Fair Scheduler to Capacity Scheduler, this Jira is created as an umbrella to track all related efforts for the migration, the scope contains

      • Bug fixes
      • Add missing features
      • Migration tools that help to generate CS configs based on FS, validate configs etc
      • Documents

      this is part of CS component, the purpose is to make the migration process smooth.

        Attachments

        1. FS-CS Migration.pdf
          75 kB
          Gergely Pollak

        Issue Links

        1.
        Support submitting apps with queue path for CapacityScheduler Sub-task Patch Available zhoukang Actions
        2.
        Migration tool that help to generate CS config based on FS config [Phase 1] Sub-task Resolved Peter Bacsko Actions
        3.
        Migration tool that help to generate CS config based on FS config [Phase 2] Sub-task Resolved Peter Bacsko Actions
        4.
        Docs about how to migrate from FS to CS config Sub-task Open Unassigned Actions
        5.
        When we expanding queue list the scheduler page will not show any applications Sub-task Open zhoukang Actions
        6.
        Document Queue and App Ordering Policy for CS Sub-task Open Prabhu Joseph Actions
        7.
        RM fails to start with FileSystemBasedConfigurationProvider Sub-task Resolved Prabhu Joseph Actions
        8.
        Allow capacity-scheduler configuration on HDFS and support reload from HDFS Sub-task Open zhoukang Actions
        9.
        Allow multiple leaf queues with the same name in CapacityScheduler Sub-task Resolved Gergely Pollak Actions
        10.
        Capacity scheduler: add support for Secondary Group rule mapping Sub-task Resolved Manikandan R Actions
        11.
        Capacity scheduler: add support for combined %user + %primary_group mapping Sub-task Resolved Manikandan R Actions
        12.
        Capacity scheduler: add support for combined %user + %secondary_group mapping Sub-task Resolved Manikandan R Actions
        13.
        Capacity scheduler: add support for limiting maxRunningApps per user Sub-task Resolved Peter Bacsko Actions
        14.
        Capacity scheduler: add support for default maxRunningApps limit per user Sub-task Resolved Peter Bacsko Actions
        15.
        Capacity scheduler: enhance capacity / maximum-capacity setting Sub-task Resolved Unassigned Actions
        16.
        Capacity scheduler: support DRF ordering policy on queue level Sub-task Patch Available Manikandan R Actions
        17.
        Capacity scheduler: enhance leaf-queue-template capacity / maximum-capacity setting Sub-task Open Manikandan R Actions
        18.
        Investigate how resource allocation configuration could be more consistent in CapacityScheduler Sub-task Open Unassigned Actions
        19.
        Fix JavaDoc errors introduced by YARN-9699 Sub-task Resolved Peter Bacsko Actions
        20.
        Support max running app logic for CapacityScheduler Sub-task Resolved Peter Bacsko Actions
        21.
        Support proxy acl user for CapacityScheduler Sub-task Open zhoukang Actions
        22.
        FairOrderingPolicy Improvements Sub-task Resolved Manikandan R Actions
        23.
        FIFOOrderingPolicy Improvements Sub-task Patch Available Manikandan R Actions
        24.
        Add dry-run feature to FS-CS converter tool Sub-task Resolved Peter Bacsko Actions
        25.
        FS-CS converter: disable terminal placement rule checking Sub-task Resolved Peter Bacsko Actions
        26.
        FS-CS converter: remove mixed ordering policy check Sub-task Resolved Peter Bacsko Actions
        27.
        FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues Sub-task Resolved Peter Bacsko Actions
        28.
        Capacity scheduler: add support for %specified mapping Sub-task Resolved Tanu Ajmera Actions
        29.
        Capacity scheduler: add support for create=true/false per mapping rule Sub-task Resolved Unassigned Actions
        30.
        FS-CS converter: dry run should work without output defined Sub-task Resolved Peter Bacsko Actions
        31.
        FS-CS converter: separator between mapping rules should be comma Sub-task Resolved Peter Bacsko Actions
        32.
        FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping Sub-task Resolved Gergely Pollak Actions
        33.
        FS-CS converter: adjust queue ACL to have the same output with CS as for FS has Sub-task Resolved Unassigned Actions
        34.
        FSQueueConverter should not set App Ordering Policy to Parent Queue Sub-task Resolved Peter Bacsko Actions
        35.
        FS-CS converter: Do not allow output dir to be the same as input dir Sub-task Resolved Adam Antal Actions
        36.
        FS-CS converter tool: issue warning on dynamic auto-create mapping rules Sub-task Resolved Peter Bacsko Actions
        37.
        FS-CS converter: initPropertyActions() is not called without rules file Sub-task Resolved Peter Bacsko Actions
        38.
        FS-CS converter: convert property yarn.scheduler.fair.update-interval-ms Sub-task Resolved Peter Bacsko Actions
        39.
        FS-CS Converter: Need to validate c-s.xml after converting Sub-task Resolved Peter Bacsko Actions
        40.
        FS-CS Converter: tool doesn't handle min/max resource conversion correctly Sub-task Resolved Peter Bacsko Actions
        41.
        Support vector of capacity percentages in Capacity Scheduler configuration Sub-task Open Andras Gyori Actions
        42.
        Add support for increment-allocation of custom resource types Sub-task In Progress Qi Zhu Actions
        43.
        FS-CS converter: only convert placement rules if a cmd line switch is defined Sub-task Resolved Peter Bacsko Actions
        44.
        FS-CS converter: call System.exit function call for every code path in main method Sub-task Resolved Peter Bacsko Actions
        45.
        FS-CS converter: fix incorrect capacity conversion Sub-task Resolved Peter Bacsko Actions
        46.
        FS-CS converter: fix emitted ordering policy string and max-am-resource percent value Sub-task Resolved Peter Bacsko Actions
        47.
        FS-CS converter: don't enable auto-create queue property for root Sub-task Resolved Peter Bacsko Actions
        48.
        FS-CS converter: skip increment properties for mem/vcores and fix DRF check Sub-task Resolved Peter Bacsko Actions
        49.
        FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler Sub-task Resolved Benjamin Teke Actions
        50.
        FS-CS converter: convert maxAppsDefault, maxRunningApps settings Sub-task Resolved Peter Bacsko Actions
        51.
        Document max-parallel-apps for Capacity Scheduler Sub-task Resolved Peter Bacsko Actions
        52.
        FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler Sub-task Resolved Peter Bacsko Actions
        53.
        Remove "experimental" warning message from fs2cs Sub-task Resolved Peter Bacsko Actions
        54.
        We should support fs to cs support for auto refresh queues when conf changed, after YARN-10623 finished. Sub-task Patch Available Qi Zhu Actions
        55.
        Add document for YARN-10623 auto refresh queue conf in cs. Sub-task Patch Available Qi Zhu Actions

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              Unassigned
              Reporter:
              cheersyang Weiwei Yang

              Dates

              • Created:
                Updated:

                Issue deployment