[YARN-8933] [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Fix Version/s: 2.10.0, 3.3.0
• Component/s: amrmproxy, federation
• Assignee: Botong Huang
• Reporter: Botong Huang
• Created: 22/Oct/18 22:18
• Updated: 11/Nov/18 19:28
• Resolved: 11/Nov/18 19:13

Description

After YARN-8696, the allocate response by FederationInterceptor is merged from the responses from a random subset of all sub-clusters, depending on the async heartbeat timing. As a result, cluster-wide information fields in the response, e.g. AvailableResources and NumClusterNodes, are not consistent at all. It can even be null/zero because the specific response is merged from an empty set of sub-cluster responses.

In this patch, we let FederationInterceptor remember the last allocate response from all known sub-clusters, and always construct the cluster-wide info fields from all of them. We also moved sub-cluster timeout from LocalityMulticastAMRMProxyPolicy to FederationInterceptor, so that sub-clusters that expired (haven't had a successful allocate response for a while) won't be included in the computation.