[YARN-10210] Add a RMFailoverProxyProvider that does DNS resolution on failover

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Fix Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Roger Liu
• Reporter: Roger Liu
• Created: 03/Sep/19 23:32
• Updated: 03/Jun/21 06:41
• Resolved: 26/Mar/20 15:15

Description

In Kubernetes, the a node may go down and then come back later with a different IP address. YARN clients which are already running will be unable to rediscover the node after it comes back up due to caching the original IP address. This is problematic for cases such as Spark HA on Kubernetes, as the node containing the resource manager may go down and come back up, meaning existing node managers must then also be restarted.

Issue Links

• causes: YARN-10803 [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException
• depends upon: HADOOP-16938 Make non-HA proxy providers pluggable
• links to: GitHub Pull Request #1399
• links to: GitHub Pull Request #1915