Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7402

Federation V2: Global Optimizations

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • federation

    Description

      YARN Federation today requires manual configuration of queues within each sub-cluster, and each RM operates "in isolation". This has few issues:

      1. Preemption is computed locally (and might far exceed the global need)
      2. Jobs within a queue are forced to consume their resources "evenly" based on queue mapping

      This umbrella JIRA tracks a new feature that leverages the FederationStateStore as a synchronization mechanism among RMs, and allows for allocation and preemption decisions to be based on a (close to up-to-date) global view of the cluster allocation and demand. The JIRA also tracks algorithms to automatically generate policies for Router and AMRMProxy to shape the traffic to each sub-cluster, and general "maintenance" of the FederationStateStore.

      Attachments

        Issue Links

          1.
          [GQ] Compute global and local "IdealAllocation" Sub-task Patch Available Carlo Curino  
          2.
          [GQ] Rebalance queue configuration for load-balancing and locality affinities Sub-task Patch Available Carlo Curino  
          3.
          [GQ] propagate to GPG queue-level utilization/pending information Sub-task Open Abhishek Modi  
          4.
          [GQ] Bias container allocations based on global view Sub-task Open Abhishek Modi  
          5.
          [GQ] Generator for queue hierarchies over federated clusters Sub-task Open Konstantinos Karanasos  
          6.
          [GQ] Compare resource allocation achieved by rebalancing algorithms with single-cluster capacity scheduler allocation Sub-task Open Konstantinos Karanasos  
          7.
          [PERF/TEST] Extend SLS to support simulation of a Federated Environment Sub-task Patch Available Shilun Fan  
          8.
          [PERF/TEST] Performance testing of CapacityScheudler at many-thousands of queues Sub-task Open Abhishek Modi  
          9.
          [PERF/TEST] Performance testing of ReservationSystem at high job submission rates Sub-task Open Xiaohua (Victor) Liang  
          10.
          [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan Operations Sub-task Patch Available Xiaohua (Victor) Liang  
          11.
          [FederationStateStore] Create a proxy chain for FederationStateStore API in the Router Sub-task Patch Available Shilun Fan

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          12.
          [FederationStateStore - MySql] Improve ApplicationHome Mysql Script. Sub-task Patch Available Shilun Fan  
          13.
          [GPG] YARN GPG mistakenly deleted applicationid Sub-task In Progress Shilun Fan

          0%

          Original Estimate - 168h
          Remaining Estimate - 168h
          14.
          Federation "Capacity Allocation" across sub-cluster Sub-task In Progress Jiandan Yang  
          15.
          New ResourceCalculator implementation that operates on vector of resources, but respect sum of ratios Sub-task In Progress Jiandan Yang  
          16.
          Add colored policies to enable manual load balancing across sub clusters Sub-task Open Chenyu Zheng  

          Activity

            People

              curino Carlo Curino
              curino Carlo Curino
              Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 168h Original Estimate - 168h
                  168h
                  Remaining:
                  Remaining Estimate - 168h
                  168h
                  Logged:
                  Remaining Estimate - 168h
                  20m