[YARN-7402] Federation V2: Global Optimizations

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: federation
• Labels: pull-request-available
• Assignee: Carlo Curino
• Reporter: Carlo Curino
• Created: 27/Oct/17 00:06
• Updated: 28/Oct/23 02:19

Description

YARN Federation today requires manual configuration of queues within each sub-cluster, and each RM operates "in isolation". This has few issues:

1. Preemption is computed locally (and might far exceed the global need)
2. Jobs within a queue are forced to consume their resources "evenly" based on queue mapping

This umbrella JIRA tracks a new feature that leverages the FederationStateStore as a synchronization mechanism among RMs, and allows for allocation and preemption decisions to be based on a (close to up-to-date) global view of the cluster allocation and demand. The JIRA also tracks algorithms to automatically generate policies for Router and AMRMProxy to shape the traffic to each sub-cluster, and general "maintenance" of the FederationStateStore.

Issue Links

• is related to: YARN-5597 YARN Federation improvements
• is related to: YARN-11105 Prohibit jobs from running across DCs by YARN Federation
• links to: GitHub Pull Request #5901

Sub-Tasks

• [GQ] Compute global and local "IdealAllocation": Patch Available
• [GQ] Rebalance queue configuration for load-balancing and locality affinities: Patch Available
• [GQ] propagate to GPG queue-level utilization/pending information: Open
• [GQ] Bias container allocations based on global view: Open
• [GQ] Generator for queue hierarchies over federated clusters: Open
• [GQ] Compare resource allocation achieved by rebalancing algorithms with single-cluster capacity scheduler allocation: Open
• [PERF/TEST] Extend SLS to support simulation of a Federated Environment: Patch Available
• [PERF/TEST] Performance testing of CapacityScheudler at many-thousands of queues: Open
• [PERF/TEST] Performance testing of ReservationSystem at high job submission rates: Open
• [PERF/TEST] Add JMX metrics for Plan Follower and Agent Placement and Plan Operations: Patch Available
• [FederationStateStore] Create a proxy chain for FederationStateStore API in the Router: Patch Available
• [FederationStateStore - MySql] Improve ApplicationHome Mysql Script.: Patch Available
• [GPG] YARN GPG mistakenly deleted applicationid: In Progress
• Federation "Capacity Allocation" across sub-cluster: In Progress
• New ResourceCalculator implementation that operates on vector of resources, but respect sum of ratios: In Progress
• Add colored policies to enable manual load balancing across sub clusters: Open