• Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: scheduler
    • Labels:


      Yarn workload management can be little more dynamic.

      1. Create yarn pool by specifying more than one Secondary AD group.

      In a multi-tenant cluster there can be hundreds of AD groups per tenant and hundreds of users per AD group. We want a way to group like workloads into single yarn pool by specifying multiple secondary AD Groups.

      Ex: All the ETL workloads of tenants needs to go into one yarn pool. This requires addition of all ETL related AD groups into one yarn pool.

      2. Demotions

      Scenario: A particular workload/job has been started in a high priority yarn pool based on the assumption that it would finish quickly but due to some data issue/change in the code/query etc. - now it is running longer and consuming high amounts of resources for long time. In this case we want to demote this to a lower resource allocated yarn pool. We don’t want this one run-away workload/job to dominate the cluster because our assumption was wrong.

      Ex: If any workload in the yarn pool runs for X minutes and/or consumes Y resources either alert me or push to another yarn pool. User can keep demoting and can push to a yarn pool which has capped resources - like Penalty box.


          Issue Links



              • Assignee:
                AjaiOmtri Ajai Omtri
              • Votes:
                1 Vote for this issue
                5 Start watching this issue


                • Created: