[YARN-7402] Federation V2: Global Optimizations

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: federation
• Labels: pull-request-available
• Assignee: Carlo Curino
• Reporter: Carlo Curino
• Created: 27/Oct/17 00:06
• Updated: 28/Oct/23 02:19

Description

YARN Federation today requires manual configuration of queues within each sub-cluster, and each RM operates "in isolation". This has few issues:

1. Preemption is computed locally (and might far exceed the global need)
2. Jobs within a queue are forced to consume their resources "evenly" based on queue mapping

This umbrella JIRA tracks a new feature that leverages the FederationStateStore as a synchronization mechanism among RMs, and allows for allocation and preemption decisions to be based on a (close to up-to-date) global view of the cluster allocation and demand. The JIRA also tracks algorithms to automatically generate policies for Router and AMRMProxy to shape the traffic to each sub-cluster, and general "maintenance" of the FederationStateStore.

Issue Links

• is related to: YARN-5597 YARN Federation improvements
• is related to: YARN-11105 Prohibit jobs from running across DCs by YARN Federation
• links to: GitHub Pull Request #5901

Sub-Tasks

• [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos: Resolved
• [GQ] Data structures for federation global queues calculations: Resolved
• [GQ] Compute global and local "IdealAllocation": Patch Available
• [GQ] Compute global "ideal allocation" including locality biases: Resolved
• [GQ] Rebalance queue configuration for load-balancing and locality affinities: Patch Available
• [GQ] propagate to GPG queue-level utilization/pending information: Open
• [GQ] Bias container allocations based on global view: Open
• [GQ] Generator for queue hierarchies over federated clusters: Open
• [GQ] Compare resource allocation achieved by rebalancing algorithms with single-cluster capacity scheduler allocation: Open
• [RESERVATION] Support ListReservation APIs in Federation Router: Resolved
• [GPG] Federation Global Policy Generator (service hook only): Resolved
• [GPG] Add SubClusterCleaner in Global Policy Generator: Resolved
• [GPG] ApplicationCleaner in Global Policy Generator: Resolved
• [GPG] Policy generator framework: Resolved
• [GPG] Load based policy generator: Resolved
• [PERF/TEST] Extend SLS to support simulation of a Federated Environment: Patch Available
• [PERF/TEST] Performance testing of CapacityScheudler at many-thousands of queues: Open
• [PERF/TEST] Performance testing of ReservationSystem at high job submission rates: Open
• [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan Operations: Patch Available
• [FederationStateStore] Create a proxy chain for FederationStateStore API in the Router: Patch Available
• [AMRMProxy] Stateful FederationInterceptor for pending requests: Resolved
• [AMRMProxy] AMRMClientRelayer for stateful FederationInterceptor: Resolved
• [Router] Federation: routing getAppState REST invocations transparently to multiple RMs: Resolved
• [Router] Federation: Improve Router REST API Metrics: Resolved
• [GPG] Fix potential connection leak in GPGUtils: Resolved
• [Router] Implement missing FederationClientInterceptor#getApplications(): Resolved
• [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.: Patch Available
• [Router] Add cache for fast answers to getApps: Resolved
• [GPG] Add max heap config option for Federation GPG: Resolved
• [GPG] Add FederationStateStore getAppInfo API for GlobalPolicyGenerator: Resolved
• [GPG] Add Yarn Registry cleanup in ApplicationCleaner: Resolved
• [GPG] Add JvmMetricsInfo and pause monitor: Resolved
• [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner: Resolved
• [GPG] Support HTTPS in GPG: Resolved
• [GPG] Support Secure Mode: Resolved
• [GPG] YARN GPG mistakenly deleted applicationid: In Progress
• Federation "Capacity Allocation" across sub-cluster: In Progress
• New ResourceCalculator implementation that operates on vector of resources, but respect sum of ratios: In Progress
• [Federation] GPG Support Query Policies In Web.: Resolved
• Add YARN_GLOBALPOLICYGENERATOR_HEAPSIZE to yarn-env for GPG: Resolved
• [GPG] Add Information About YARN GPG in Federation.md: Resolved
• Add colored policies to enable manual load balancing across sub clusters: Open
• [GPG] Improve GPGPolicyFacade#getPolicyManager: Resolved
• [GPG] Improve GPGOverviewBlock Infomation: Resolved
• [GPG] Add GPGWebServices: Resolved
• [Doc] Add allow-partial-result description to Yarn Federation documentation: Resolved
• In Federation, kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM: Resolved
• [GPG] GPG Support CLI.: Resolved