[YARN-11122] Support getClusterNodes API in FederationClientInterceptor

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.3.1, 3.3.2
• Fix Version/s: 3.4.0
• Component/s: federation
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Shilun Fan
• Reporter: Shilun Fan
• Created: 02/May/22 07:15
• Updated: 25/Jan/24 15:09
• Resolved: 15/May/22 16:16

Description

Yarn Federation is a very useful feature, especially important in clusters with more than 1,000 units. For example, we have 2 offline clusters, which are used for ad-hoc and offline scheduling (etl) respectively. We hope to be able to isolate and rationally use 2 The resources of a cluster, at night, adhoc resources can be used by the etl cluster; at the same time, during breakfast (9:00-22:00), the etl cluster resources can also be used by the adhoc cluster.

At the same time, so that more people can use this feature, gradually improve the method that has not been implemented.

In YARN-10465, some methods have been implemented, from a personal point of view, some doubts.
Question 1: Without considering the metric implementation, it is impossible to understand the execution of related functions.
Question 2: Using multi-threading and reflection implementation, the readability of the related logic is relatively poor, and there is not much performance difference between this method and the conventional loop method to obtain the theory.
Question 3: The code is already 2 years old, merged into the local branch, there may be conflicts.

Issue Links

• Dependent: HADOOP-18222 Prevent DelegationTokenSecretManagerMetrics from registering multiple times
• is caused by: HADOOP-18167 Add metrics to track delegation token secret manager operations
• relates to: YARN-10465 Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation
• links to: GitHub Pull Request #4274