Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7402

Federation V2: Global Optimizations

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: federation
    • Labels:
      None

      Description

      YARN Federation today requires manual configuration of queues within each sub-cluster, and each RM operates "in isolation". This has few issues:

      1. Preemption is computed locally (and might far exceed the global need)
      2. Jobs within a queue are forced to consume their resources "evenly" based on queue mapping

      This umbrella JIRA tracks a new feature that leverages the FederationStateStore as a synchronization mechanism among RMs, and allows for allocation and preemption decisions to be based on a (close to up-to-date) global view of the cluster allocation and demand. The JIRA also tracks algorithms to automatically generate policies for Router and AMRMProxy to shape the traffic to each sub-cluster, and general "maintenance" of the FederationStateStore.

        Attachments

          Issue Links

          1.
          [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos Sub-task Resolved Carlo Curino
          2.
          [GQ] Data structures for federation global queues calculations Sub-task Resolved Abhishek Modi
          3.
          [GQ] Compute global and local "IdealAllocation" Sub-task Patch Available Carlo Curino
          4.
          [GQ] Compute global "ideal allocation" including locality biases Sub-task Resolved Carlo Curino
          5.
          [GQ] Rebalance queue configuration for load-balancing and locality affinities Sub-task Patch Available Carlo Curino
          6.
          [GQ] propagate to GPG queue-level utilization/pending information Sub-task Open Abhishek Modi
          7.
          [GQ] Bias container allocations based on global view Sub-task Open Abhishek Modi
          8.
          [GQ] Generator for queue hierarchies over federated clusters Sub-task Open Konstantinos Karanasos
          9.
          [GQ] Compare resource allocation achieved by rebalancing algorithms with single-cluster capacity scheduler allocation Sub-task Open Konstantinos Karanasos
          10.
          [RESERVATION] Support Reservation APIs in Federation Router Sub-task Open Unassigned
          11.
          [RESERVATION] Federation StateStore: support storage/retrieval of reservations Sub-task Open Unassigned
          12.
          [RESERVATION] Add support for reservation-based routing. Sub-task Patch Available Carlo Curino
          13.
          [GPG] Federation Global Policy Generator (service hook only) Sub-task Resolved Botong Huang
          14.
          [GPG] Add SubClusterCleaner in Global Policy Generator Sub-task Resolved Botong Huang
          15.
          [GPG] ApplicationCleaner in Global Policy Generator Sub-task Resolved Botong Huang
          16.
          [GPG] Policy generator framework Sub-task Resolved Young Chen
          17.
          [GPG] Load based policy generator Sub-task Resolved Young Chen
          18.
          [PERF/TEST] Extend SLS to support simulation of a Federated Environment Sub-task Patch Available Tanuj Nayak
          19.
          [PERF/TEST] Performance testing of CapacityScheudler at many-thousands of queues Sub-task Open Abhishek Modi
          20.
          [PERF/TEST] Performance testing of ReservationSystem at high job submission rates Sub-task Open Xiaohua (Victor) Liang
          21.
          [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan Operations Sub-task Patch Available Xiaohua (Victor) Liang
          22.
          [FederationStateStore] Create a proxy chain for FederationStateStore API in the Router Sub-task Patch Available Giovanni Matteo Fumarola
          23.
          [AMRMProxy] Stateful FederationInterceptor for pending requests Sub-task Resolved Botong Huang
          24.
          [AMRMProxy] AMRMClientRelayer for stateful FederationInterceptor Sub-task Resolved Botong Huang
          25.
          [Router] Federation: routing getAppState REST invocations transparently to multiple RMs Sub-task Resolved Giovanni Matteo Fumarola
          26.
          [Router] Federation: routing some missing REST invocations transparently to multiple RMs Sub-task Patch Available Yiran Wu
          27.
          [GPG] Fix potential connection leak in GPGUtils Sub-task Resolved Giovanni Matteo Fumarola
          28.
          [Router] Implement missing FederationClientInterceptor#getApplications() Sub-task Patch Available Yiran Wu
          29.
          [FederationStateStore - MySql] Deadlock In addApplicationHome Sub-task Patch Available Unassigned
          30.
          [Router] Add cache service for fast answers to getApps Sub-task Open Giovanni Matteo Fumarola
          31.
          [GPG] Add max heap config option for Federation GPG Sub-task Resolved Botong Huang
          32.
          [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator Sub-task Resolved Botong Huang
          33.
          [GPG] Add Yarn Registry cleanup in ApplicationCleaner Sub-task Resolved Botong Huang
          34.
          [GPG] Add JvmMetricsInfo and pause monitor Sub-task Resolved Bilwa S T
          35.
          [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner Sub-task Resolved Botong Huang

            Activity

              People

              • Assignee:
                curino Carlo Curino
                Reporter:
                curino Carlo Curino
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated: