[SPARK-28966] Cached DNS name resolution error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0
• Component/s: Spark Core, YARN
• Assignee: Unassigned
• Reporter: Roger Liu
• Created: 03/Sep/19 23:28
• Updated: 17/May/20 18:14
• Resolved: 03/Sep/19 23:33

Description

In Kubernetes, the a node may go down and then come back later with a different IP address. Yarn clients which are already running will be unable to rediscover the node after it comes back up due to caching the original IP address. This is problematic for cases such as Spark HA on Kubernetes, as the node containing the resource manager may go down and come back up, meaning existing node managers must then also be restarted.