Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5734

OrgQueue for easy CapacityScheduler queue configuration management

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      <!-- markdown -->

      The OrgQueue extension to the capacity scheduler provides a programmatic way to change configurations by providing a REST API that users can call to modify queue configurations. This enables automation of queue configuration management by administrators in the queue's `administer_queue` ACL.
      Show
      <!-- markdown --> The OrgQueue extension to the capacity scheduler provides a programmatic way to change configurations by providing a REST API that users can call to modify queue configurations. This enables automation of queue configuration management by administrators in the queue's `administer_queue` ACL.

      Description

      The current xml based configuration mechanism in CapacityScheduler makes it very inconvenient to apply any changes to the queue configurations. We saw 2 main drawbacks in the file based configuration mechanism:

      1. This makes it very inconvenient to automate queue configuration updates. For example, in our cluster setup, we leverage the queue mapping feature from YARN-2411 to route users to their dedicated organization queues. It could be extremely cumbersome to keep updating the config file to manage the very dynamic mapping between users to organizations.
      2. Even a user has the admin permission on one specific queue, that user is unable to make any queue configuration changes to resize the subqueues, changing queue ACLs, or creating new queues. All these operations need to be performed in a centralized manner by the cluster administrators.

      With these current limitations, we realized the need of a more flexible configuration mechanism that allows queue configurations to be stored and managed more dynamically. We developed the feature internally at LinkedIn which introduces the concept of MutableConfigurationProvider. What it essentially does is to provide a set of configuration mutation APIs that allows queue configurations to be updated externally with a set of REST APIs. When performing the queue configuration changes, the queue ACLs will be honored, which means only queue administrators can make configuration changes to a given queue. MutableConfigurationProvider is implemented as a pluggable interface, and we have one implementation of this interface which is based on Derby embedded database.

      This feature has been deployed at LinkedIn's Hadoop cluster for a year now, and have gone through several iterations of gathering feedbacks from users and improving accordingly. With this feature, cluster administrators are able to automate lots of thequeue configuration management tasks, such as setting the queue capacities to adjust cluster resources between queues based on established resource consumption patterns, or managing updating the user to queue mappings. We have attached our design documentation with this ticket and would like to receive feedbacks from the community regarding how to best integrate it with the latest version of YARN.

        Attachments

        1. OrgQueue_API-Based_Config_Management_v1.pdf
          115 kB
          Jonathan Hung
        2. OrgQueue_Design_v0.pdf
          220 kB
          Jonathan Hung
        3. OrgQueueAPI-BasedSchedulerConfigurationManagement_v2.pdf
          117 kB
          Jonathan Hung
        4. YARN-5734-YARN-5734.001.patch
          127 kB
          Jonathan Hung

          Issue Links

          1.
          Create YarnConfigurationStore interface and InMemoryConfigurationStore class Sub-task Resolved Jonathan Hung
          2.
          Create LeveldbConfigurationStore class using Leveldb as backing store Sub-task Resolved Jonathan Hung
          3.
          Implement MutableConfigurationManager for handling storage into configuration store Sub-task Resolved Jonathan Hung
          4.
          Add pluggable configuration ACL policy interface and implementation Sub-task Resolved Jonathan Hung
          5.
          Create StoreConfigurationProvider to construct a Configuration from the backing store Sub-task Resolved Unassigned
          6.
          Changes to allow CapacityScheduler to use configuration store Sub-task Resolved Jonathan Hung
          7.
          Create REST API for changing YARN scheduler configurations Sub-task Resolved Jonathan Hung
          8.
          Create CLI for changing YARN configurations Sub-task Resolved Jonathan Hung
          9.
          Implement a CapacityScheduler policy for configuration changes Sub-task Resolved Jonathan Hung
          10.
          Protocol for scheduler configuration changes between client and RM Sub-task Resolved Jonathan Hung
          11.
          Disable queue refresh when configuration mutation is enabled Sub-task Resolved Jonathan Hung
          12.
          Support global configuration mutation in MutableConfProvider Sub-task Resolved Jonathan Hung
          13.
          Support for adding and removing queue mappings Sub-task Patch Available Jonathan Hung
          14.
          Add ability to export scheduler configuration XML Sub-task Patch Available Jonathan Hung
          15.
          Implement zookeeper based store for scheduler configuration updates Sub-task Resolved Jonathan Hung
          16.
          Fix issues on recovery in LevelDB store Sub-task Resolved Jonathan Hung
          17.
          Add closing logic to configuration store Sub-task Resolved Jonathan Hung
          18.
          Documentation for API based scheduler configuration management Sub-task Resolved Jonathan Hung
          19.
          Merge YARN-5734 to trunk/branch-2 Sub-task Resolved Jonathan Hung
          20.
          Misc changes to YARN-5734 Sub-task Resolved Jonathan Hung
          21.
          Removing queue then failing over results in exception Sub-task Resolved Jonathan Hung
          22.
          Merge YARN-5734 branch to trunk branch Sub-task Resolved Xuan Gong
          23.
          Merge YARN-5734 branch to branch-3.0 Sub-task Resolved Xuan Gong
          24.
          Merge YARN-5734 branch to branch-2 Sub-task Resolved Xuan Gong
          25.
          Add file system based scheduler configuration store Sub-task Resolved Jiandan Yang

            Activity

              People

              • Assignee:
                mshen Min Shen
                Reporter:
                mshen Min Shen
              • Votes:
                2 Vote for this issue
                Watchers:
                41 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: