[YARN-5597] YARN Federation improvements

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Subramaniam Krishnan
• Reporter: Subramaniam Krishnan
• Created: 30/Aug/16 20:06
• Updated: 15/Feb/23 14:17

Description

This umbrella JIRA tracks set of improvements over the YARN Federation MVP (YARN-2915)

Issue Links

• depends upon: YARN-2915 Enable YARN RM scale out via federation using multiple RM's
• relates to: YARN-8435 Fix NPE when the same client simultaneously contact for the first time Yarn Router
• relates to: YARN-8596 Allow SQLFederationStateStore to submit the same app in the same subcluster
• relates to: YARN-8434 Update federation documentation of Nodemanager configurations
• relates to: YARN-7402 Federation V2: Global Optimizations
• relates to: YARN-10885 Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
• relates to: YARN-11333 Federation: Improve Yarn Router Web Page

Sub-Tasks

• Federation maintenance mechanisms (simple CLI and command propagation): Resolved
• Federation "Capacity Allocation" across sub-cluster: Open
• Add versioning for FederationStateStore: Resolved
• New ResourceCalculator implementation that operates on vector of resources, but respect sum of ratios: In Progress
• Add support for AMRMProxy HA: Resolved
• Advanced Federation UI based on YARN UI v2: Patch Available
• Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call: Resolved
• Consider running RM tests against the Router: In Progress
• Refactoring Router services to use common util classes for pipeline creations: Resolved
• Create SecureLogin inside Router: Resolved
• Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests: Resolved
• Handle containerId duplicate without failing the heartbeat in Federation Interceptor: Resolved
• Federation Router (hiding multiple RMs for ApplicationClientProtocol) phase 2: In Progress
• Metrics for Federation AMRMProxy: Resolved
• Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService: Resolved
• Adding RM ClusterId in AppInfo: Resolved
• Adding RM Cluster Id in ApplicationReport: Resolved
• Federation Router Web Service fixes: Resolved
• Race condition between second app attempt and UAM timeout when first attempt node is down: Patch Available
• Handle AM register requests asynchronously in FederationInterceptor: Resolved
• Add config in FederationRMFailoverProxy to not bypass facade cache when failing over: Resolved
• AMRMProxy recover should catch for all throwable to avoid premature exit: Resolved
• Yarn RM Epoch should wrap around: Resolved
• [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy: Resolved
• [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor: Resolved
• [AMRMProxy] More robust responseId resync after an YarnRM master slave switch: Resolved
• [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async: Resolved
• LocalityMulticastAMRMProxyPolicy should fallback to random sub-cluster when cannot resolve resource: Resolved
• Add Yarnclient#yarnclusterMetrics API implementation in router: Resolved
• Refactor the UAM heartbeat thread in preparation for YARN-8696: Resolved
• Add clean up for FederationStore apps: Resolved
• [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer: Resolved
• Add audit logs for router service: Resolved
• Create HomePolicyManager that sends all the requests to the home subcluster: Resolved
• AMRMProxyPolicies should accept heartbeat response from new/unknown subclusters: Resolved
• [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client: Resolved
• Fix FederationInterceptor#allocate to set application priority in allocateResponse: In Progress
• [Router] Federation: routing getContainers REST invocations transparently to multiple RMs: Resolved
• [Router] Add JvmMetricsInfo and pause monitor: Resolved
• [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor: Resolved
• [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size: Resolved
• [Router] Add missing methods in RMWebProtocol: Resolved
• Mapreduce application container start fail after AM restart.: In Progress
• [Router] Add locality policy: Resolved
• Add znode hierarchy in Federation ZK State Store: In Progress
• Add application submit data to state store: In Progress
• Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest: Resolved
• Add colored policies to enable manual load balancing across sub clusters: Open
• Make AMRMProxyPolicy aware of SC load: Patch Available
• In Federation Secure cluster Application submission fails when authorization is enabled: Resolved
• In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled: Resolved
• [Router] Router Audit Log Add Client IP Address.: Resolved
• Metrics for Federation getClusterMetrics: Resolved
• Router's main() should support generic options: Open
• [Router] UGI conf doesn't read user overridden configurations on Router startup: Open
• [Router] FederationStateStoreFacade is not reinitialized with Router conf: Open