[YARN-8696] [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.2.0
• Component/s: nodemanager
• Assignee: Botong Huang
• Reporter: Botong Huang
• Created: 21/Aug/18 21:24
• Updated: 15/Oct/18 16:45
• Resolved: 24/Sep/18 18:42

Description

Today in FederationInterceptor, the heartbeat to home sub-cluster is synchronous. After the heartbeat is sent out to home sub-cluster, it waits for the home response to come back before merging and returning the (merged) heartbeat result to back AM. If home sub-cluster is suffering from connection issues, or down during an YarnRM master-slave switch, all heartbeat threads in FederationInterceptor will be blocked waiting for home response. As a result, the successful UAM heartbeats from secondary sub-clusters will not be returned to AM at all. Additionally, because of the fact that we kept the same heartbeat responseId between AM and home RM, lots of tricky handling are needed regarding the responseId resync when it comes to FederationInterceptor (part of AMRMProxy, NM) work preserving restart (YARN-6127, YARN-1336), home RM master-slave switch etc.

In this patch, we change the heartbeat to home sub-cluster to asynchronous, same as the way we handle UAM heartbeats in secondaries. So that any sub-cluster down or connection issues won't impact AM getting responses from other sub-clusters. The responseId is also managed separately for home sub-cluster and AM, and they increment independently. The resync logic becomes much cleaner.

Issue Links

• depends upon: YARN-8705 Refactor the UAM heartbeat thread in preparation for YARN-8696