Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698

[Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: capacity scheduler
    • Labels:

      Description

      We see some users want to migrate from Fair Scheduler to Capacity Scheduler, this Jira is created as an umbrella to track all related efforts for the migration, the scope contains

      • Bug fixes
      • Add missing features
      • Migration tools that help to generate CS configs based on FS, validate configs etc
      • Documents

      this is part of CS component, the purpose is to make the migration process smooth.

        Attachments

        1. FS-CS Migration.pdf
          75 kB
          Gergely Pollak

          Issue Links

          1.
          Support submitting apps with queue path for CapacityScheduler Sub-task Patch Available zhoukang
          2.
          Migration tool that help to generate CS config based on FS config [Phase 1] Sub-task Resolved Peter Bacsko
          3.
          Migration tool that help to generate CS config based on FS config [Phase 2] Sub-task Resolved Peter Bacsko
          4.
          Docs about how to migrate from FS to CS config Sub-task Open Unassigned
          5.
          When we expanding queue list the scheduler page will not show any applications Sub-task Open zhoukang
          6.
          Document Queue and App Ordering Policy for CS Sub-task Open Prabhu Joseph
          7.
          RM fails to start with FileSystemBasedConfigurationProvider Sub-task Resolved Prabhu Joseph
          8.
          Allow capacity-scheduler configuration on HDFS and support reload from HDFS Sub-task Open zhoukang
          9.
          Allow multiple leaf queues with the same name in CapacityScheduler Sub-task Resolved Gergely Pollak
          10.
          Capacity scheduler: add support for Secondary Group rule mapping Sub-task Resolved Manikandan R
          11.
          Capacity scheduler: add support for combined %user + %primary_group mapping Sub-task Resolved Manikandan R
          12.
          Capacity scheduler: add support for combined %user + %secondary_group mapping Sub-task Resolved Manikandan R
          13.
          Capacity scheduler: add support for limiting maxRunningApps per user Sub-task Resolved Peter Bacsko
          14.
          Capacity scheduler: add support for default maxRunningApps limit per user Sub-task Resolved Peter Bacsko
          15.
          Capacity scheduler: enhance capacity / maximum-capacity setting Sub-task Resolved Unassigned
          16.
          Capacity scheduler: support DRF ordering policy on queue level Sub-task Patch Available Manikandan R
          17.
          Capacity scheduler: enhance leaf-queue-template capacity / maximum-capacity setting Sub-task Open Manikandan R
          18.
          Investigate how resource allocation configuration could be more consistent in CapacityScheduler Sub-task Open Unassigned
          19.
          Fix JavaDoc errors introduced by YARN-9699 Sub-task Resolved Peter Bacsko
          20.
          Support max running app logic for CapacityScheduler Sub-task Resolved Peter Bacsko
          21.
          Support proxy acl user for CapacityScheduler Sub-task Open zhoukang
          22.
          FairOrderingPolicy Improvements Sub-task Resolved Manikandan R
          23.
          FIFOOrderingPolicy Improvements Sub-task Patch Available Manikandan R
          24.
          Add dry-run feature to FS-CS converter tool Sub-task Resolved Peter Bacsko
          25.
          FS-CS converter: disable terminal placement rule checking Sub-task Resolved Peter Bacsko
          26.
          FS-CS converter: remove mixed ordering policy check Sub-task Resolved Peter Bacsko
          27.
          FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues Sub-task Resolved Peter Bacsko
          28.
          Capacity scheduler: add support for %specified mapping Sub-task Patch Available Tanu Ajmera
          29.
          Capacity scheduler: add support for create=true/false per mapping rule Sub-task Open Unassigned
          30.
          FS-CS converter: dry run should work without output defined Sub-task Resolved Peter Bacsko
          31.
          FS-CS converter: separator between mapping rules should be comma Sub-task Resolved Peter Bacsko
          32.
          FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping Sub-task Resolved Gergely Pollak
          33.
          FS-CS converter: adjust queue ACL to have the same output with CS as for FS has Sub-task Resolved Unassigned
          34.
          FSQueueConverter should not set App Ordering Policy to Parent Queue Sub-task Resolved Peter Bacsko
          35.
          FS-CS converter: Do not allow output dir to be the same as input dir Sub-task Resolved Adam Antal
          36.
          FS-CS converter tool: issue warning on dynamic auto-create mapping rules Sub-task Resolved Peter Bacsko
          37.
          FS-CS converter: initPropertyActions() is not called without rules file Sub-task Resolved Peter Bacsko
          38.
          FS-CS converter: convert property yarn.scheduler.fair.update-interval-ms Sub-task Resolved Peter Bacsko
          39.
          FS-CS Converter: Need to validate c-s.xml after converting Sub-task Resolved Peter Bacsko
          40.
          FS-CS Converter: tool doesn't handle min/max resource conversion correctly Sub-task Resolved Peter Bacsko
          41.
          Support vector of capacity percentages in Capacity Scheduler configuration Sub-task Open Zoltan Siegl
          42.
          Add support for increment-allocation of custom resource types Sub-task Open Unassigned
          43.
          FS-CS converter: only convert placement rules if a cmd line switch is defined Sub-task Resolved Peter Bacsko
          44.
          FS-CS converter: call System.exit function call for every code path in main method Sub-task Resolved Peter Bacsko
          45.
          FS-CS converter: fix incorrect capacity conversion Sub-task Resolved Peter Bacsko
          46.
          FS-CS converter: fix emitted ordering policy string and max-am-resource percent value Sub-task Resolved Peter Bacsko
          47.
          FS-CS converter: don't enable auto-create queue property for root Sub-task Resolved Peter Bacsko
          48.
          FS-CS converter: skip increment properties for mem/vcores and fix DRF check Sub-task Resolved Peter Bacsko
          49.
          FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler Sub-task Resolved Benjamin Teke
          50.
          FS-CS converter: convert maxAppsDefault, maxRunningApps settings Sub-task Resolved Peter Bacsko
          51.
          Document max-parallel-apps for Capacity Scheduler Sub-task Resolved Peter Bacsko

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cheersyang Weiwei Yang
              • Votes:
                1 Vote for this issue
                Watchers:
                25 Start watching this issue

                Dates

                • Created:
                  Updated: