[YARN-5597] YARN Federation improvements

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: federation
• Labels: federation
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed

Release Note

We have enhanced the YARN Federation functionality for improved usability. The enhanced features are as follows:
1. YARN Router now boasts a full implementation of all interfaces including the ApplicationClientProtocol, ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
2. YARN Router support for application cleanup and automatic offline mechanisms for subCluster.
3. Code improvements were undertaken for the Router and AMRMProxy, along with enhancements to previously pending functionalities.
4. Audit logs and Metrics for Router received upgrades.
5. A boost in cluster security features was achieved, with the inclusion of Kerberos support.
6. The page function of the router has been enhanced.
7. A set of commands has been added to the Router side for operating on SubClusters and Policies.

• Flags: Important
• Assignee: Subramaniam Krishnan
• Reporter: Subramaniam Krishnan
• Created: 30/Aug/16 20:06
• Updated: 25/Jan/24 23:26
• Resolved: 05/Dec/23 17:31

Description

This umbrella JIRA tracks set of improvements over the YARN Federation MVP (YARN-2915)

Issue Links

• depends upon: YARN-2915 Enable YARN RM scale out via federation using multiple RM's
• is related to: YARN-10229 [Federation] Client should be able to submit application to RM directly using normal client conf
• is related to: YARN-10900 Yarn nodes missing from router web app
• is related to: YARN-9708 Yarn Router Support DelegationToken
• is related to: YARN-9586 [QA] Need more doc for yarn.federation.policy-manager-params when LoadBasedRouterPolicy is used
• relates to: YARN-8435 Fix NPE when the same client simultaneously contact for the first time Yarn Router
• relates to: YARN-8596 Allow SQLFederationStateStore to submit the same app in the same subcluster
• relates to: YARN-8434 Update federation documentation of Nodemanager configurations
• relates to: YARN-7402 Federation V2: Global Optimizations
• relates to: YARN-10885 Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
• relates to: YARN-11333 Federation: Improve Yarn Router Web Page

Sub-Tasks

• Federation maintenance mechanisms (simple CLI and command propagation): Resolved
• Add versioning for FederationStateStore: Resolved
• Add support for AMRMProxy HA: Resolved
• Advanced Federation UI based on YARN UI v2: Resolved
• Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call: Resolved
• Running RM tests against the Router: Resolved
• Refactoring Router services to use common util classes for pipeline creations: Resolved
• Create SecureLogin inside Router: Resolved
• Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests: Resolved
• Handle containerId duplicate without failing the heartbeat in Federation Interceptor: Resolved
• Federation Router (hiding multiple RMs for ApplicationClientProtocol) phase 2: Resolved
• Metrics for Federation AMRMProxy: Resolved
• Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService: Resolved
• Adding RM ClusterId in AppInfo: Resolved
• Adding RM Cluster Id in ApplicationReport: Resolved
• Federation Router Web Service fixes: Resolved
• Race condition between second app attempt and UAM timeout when first attempt node is down: Resolved
• Handle AM register requests asynchronously in FederationInterceptor: Resolved
• Add config in FederationRMFailoverProxy to not bypass facade cache when failing over: Resolved
• AMRMProxy recover should catch for all throwable to avoid premature exit: Resolved
• Yarn RM Epoch should wrap around: Resolved
• [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy: Resolved
• [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor: Resolved
• [AMRMProxy] More robust responseId resync after an YarnRM master slave switch: Resolved
• [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async: Resolved
• LocalityMulticastAMRMProxyPolicy should fallback to random sub-cluster when cannot resolve resource: Resolved
• Add Yarnclient#yarnclusterMetrics API implementation in router: Resolved
• Refactor the UAM heartbeat thread in preparation for YARN-8696: Resolved
• Add clean up for FederationStore apps: Resolved
• [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer: Resolved
• Add audit logs for router service: Resolved
• Create HomePolicyManager that sends all the requests to the home subcluster: Resolved
• AMRMProxyPolicies should accept heartbeat response from new/unknown subclusters: Resolved
• [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client: Resolved
• Fix FederationInterceptor#allocate to set application priority in allocateResponse: Resolved
• [Router] Federation: routing getContainers REST invocations transparently to multiple RMs: Resolved
• [Router] Add JvmMetricsInfo and pause monitor: Resolved
• [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor: Resolved
• [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size: Resolved
• [Router] Add missing methods in RMWebProtocol: Resolved
• Mapreduce application container start fail after AM restart.: Resolved
• [Router] Add locality policy: Resolved
• Add znode hierarchy in Federation ZK State Store: Resolved
• Add application submit data to state store: Resolved
• Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest: Resolved
• Make AMRMProxyPolicy aware of SC load: Resolved
• In Federation Secure cluster Application submission fails when authorization is enabled: Resolved
• In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled: Resolved
• [Router] Router Audit Log Add Client IP Address.: Resolved
• Metrics for Federation getClusterMetrics: Resolved
• Router's main() should support generic options: Resolved
• [Router] UGI conf doesn't read user overridden configurations on Router startup: Resolved
• [Router] FederationStateStoreFacade is not reinitialized with Router conf: Resolved
• Make proxy server support YARN federation.: Resolved
• Make router support proxy server.: Resolved
• [Federation] Client should be able to submit application to RM directly using normal client conf: Resolved
• [Federation] Add FederationInterceptor#allow-partial-result config.: Resolved
• Improve FederationInterceptorREST#createInterceptorForSubCluster Use WebAppUtils: Resolved
• [Federation] Fix some PBImpl classes to avoid NPE.: Resolved
• [Federation] Improve SubClusterState#fromString parameter and LogMessage: Resolved
• Improve equals, hashCode(), toString() methods of the Federation Base Object: Resolved
• In federation and security mode, nm recover may fail.: Resolved
• [Federation] Add SQLServer Script and Supported DB Version in Federation.md: Resolved
• [Federation] Improve FederationClientInterceptor#ThreadPool thread pool configuration.: Resolved
• PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value: Resolved
• Fix token reset synchronization for UAM response token: Resolved
• Router Page display the db username and password in mask mode: Resolved
• Add enhanced headroom in AllocateResponse: Resolved
• Add YARN_ROUTER_HEAPSIZE to yarn-env for routers: Resolved
• [Federation] Fix Yarn federation.md table format: Resolved
• Fix Yarn Router Broken Link: Resolved
• [Federation] Fix Typo of NodeManager AMRMProxy.: Resolved
• Improve Yarn Router Junit Test Close MockRM: Resolved
• [Federation] Improve NM FederationInterceptor removeAppFromRegistry: Resolved
• [Federation] Improve UnmanagedAMPoolManager WithoutBlock ServiceStop: Resolved
• [Federation] Add batchFinishApplicationMaster in UAMPoolManager: Resolved
• Capture the Performance Metrics of ZookeeperFederationStateStore: Resolved
• [Federation] ConfiguredRMFailoverProxyProvider Supports Randomly Select an Router.: Resolved
• Improve FederationInterceptorREST AuditLog: Resolved
• Improve FederationInterceptorREST Method Result: Resolved
• [Federation] Add WeightedHomePolicyManager: Resolved
• Make YARN Router throw Exception to client clearly: Resolved
• [Federation] Improve Yarn Federation documentation: Resolved
• Fix uncleaned threads in YARN Federation interceptor threadpool: Resolved
• Fix typos in hadoop-yarn-server-common#federation: Resolved
• Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data: Resolved
• YARN Router Web supports displaying information for Non-Federation.: Resolved
• [Minor] Improve UnmanagedAMPoolManager/UnmanagedApplicationManager Code: Resolved
• [Federation] Code cleanup for NodeManager#amrmproxy: Resolved
• [Federation] Improve DefaultRequestInterceptor#init Code: Resolved
• The FederationInterceptor#launchUAM Added retry logic.: Resolved
• Improve the Policy Description in Federation.md: Resolved
• [Federation] Add RouterAuditLog to log4j.properties: Resolved
• [Federation] Fix NodeManager#TestFederationInterceptor Flaky Unit Test: Resolved
• Improve existsApplicationHomeSubCluster/existsReservationHomeSubCluster Log Level: Resolved
• [Federation] Add Steps To Set up a Test Cluster.: Resolved
• Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster: Resolved
• Improve createJerseyClient#setConnectTimeout Code: Resolved
• [Federation] SQLFederationStateStore Support Store ApplicationSubmitData: Resolved
• [Federation] Improve FederationClientInterceptor To Return Partial Results of subClusters.: Resolved
• Let Federation.md more standardized: Resolved
• Change the time unit of scCleanerIntervalMs in Router: Resolved
• Improve the time unit for FederationRMAdminInterceptor#heartbeatExpirationMillis: Resolved