Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9698

[Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: capacity scheduler
    • Labels:

      Description

      We see some users want to migrate from Fair Scheduler to Capacity Scheduler, this Jira is created as an umbrella to track all related efforts for the migration, the scope contains

      • Bug fixes
      • Add missing features
      • Migration tools that help to generate CS configs based on FS, validate configs etc
      • Documents

      this is part of CS component, the purpose is to make the migration process smooth.

        Attachments

        1. FS-CS Migration.pdf
          75 kB
          Gergely Pollak

          Issue Links

          1.
          Support submitting apps with queue path for CapacityScheduler Sub-task Patch Available zhoukang
          2.
          Migration tool that help to generate CS config based on FS config [Phase 1] Sub-task Resolved Peter Bacsko
          3.
          Migration tool that help to generate CS config based on FS config [Phase 2] Sub-task Resolved Peter Bacsko
          4.
          Docs about how to migrate from FS to CS config Sub-task Open Unassigned
          5.
          When we expanding queue list the scheduler page will not show any applications Sub-task Open zhoukang
          6.
          Document Queue and App Ordering Policy for CS Sub-task Open Prabhu Joseph
          7.
          RM fails to start with FileSystemBasedConfigurationProvider Sub-task Resolved Prabhu Joseph
          8.
          Allow capacity-scheduler configuration on HDFS and support reload from HDFS Sub-task Open zhoukang
          9.
          Allow multiple leaf queues with the same name in CS Sub-task Patch Available Gergely Pollak
          10.
          Capacity scheduler: add support for Secondary Group rule mapping Sub-task Resolved Manikandan R
          11.
          Capacity scheduler: add support for combined %user + %primary_group mapping Sub-task Resolved Manikandan R
          12.
          Capacity scheduler: add support for combined %user + %secondary_group mapping Sub-task Resolved Manikandan R
          13.
          Capacity scheduler: add support for limiting maxRunningApps per user Sub-task Open Peter Bacsko
          14.
          Capacity scheduler: add support for default maxRunningApps limit per user Sub-task Open Peter Bacsko
          15.
          Capacity scheduler: enhance capacity / maximum-capacity setting Sub-task Resolved Unassigned
          16.
          Capacity scheduler: support DRF ordering policy on queue level Sub-task Patch Available Manikandan R
          17.
          Capacity scheduler: enhance leaf-queue-template capacity / maximum-capacity setting Sub-task Open Peter Bacsko
          18.
          Investigate how resource allocation configuration could be more consistent in CapacityScheduler Sub-task Open Unassigned
          19.
          Fix JavaDoc errors introduced by YARN-9699 Sub-task Resolved Peter Bacsko
          20.
          Support max running app logic for CapacityScheduler Sub-task Open zhoukang
          21.
          Support proxy acl user for CapacityScheduler Sub-task Open zhoukang
          22.
          FairOrderingPolicy Improvements Sub-task Patch Available Manikandan R
          23.
          FIFOOrderingPolicy Improvements Sub-task Patch Available Manikandan R
          24.
          Add dry-run feature to FS-CS converter tool Sub-task Resolved Peter Bacsko
          25.
          FS-CS converter: disable terminal placement rule checking Sub-task Resolved Peter Bacsko
          26.
          FS-CS converter: remove mixed ordering policy check Sub-task Resolved Peter Bacsko
          27.
          FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues Sub-task Resolved Peter Bacsko
          28.
          Capacity scheduler: add support for %specified mapping Sub-task Open Unassigned
          29.
          Capacity scheduler: add support for create=true/false per mapping rule Sub-task Open Unassigned
          30.
          FS-CS converter: dry run should work without output defined Sub-task Resolved Peter Bacsko
          31.
          FS-CS converter: separator between mapping rules should be comma Sub-task Resolved Peter Bacsko
          32.
          FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping Sub-task Open Peter Bacsko
          33.
          FS-CS converter: adjust queue ACL to have the same output with CS as for FS has Sub-task Resolved Unassigned
          34.
          FSQueueConverter should not set App Ordering Policy to Parent Queue Sub-task Resolved Peter Bacsko
          35.
          FS-CS converter: Do not allow output dir to be the same as input dir Sub-task Resolved Adam Antal
          36.
          FS-CS converter tool: issue warning on dynamic auto-create mapping rules Sub-task Resolved Peter Bacsko
          37.
          FS-CS converter: initPropertyActions() is not called without rules file Sub-task Resolved Peter Bacsko
          38.
          FS-CS converter: convert property yarn.scheduler.fair.update-interval-ms Sub-task Open Peter Bacsko

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cheersyang Weiwei Yang
              • Votes:
                1 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated: