[SPARK-39473] NullPointerException in SparkRackResolver

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.8, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1
• Component/s: YARN
• Assignee: Attila Zsolt Piros
• Reporter: Attila Zsolt Piros
• Created: 14/Jun/22 22:22
• Updated: 14/Jun/22 22:47
• Resolved: 14/Jun/22 22:47

Description

When the underlying DNSToSwitchMapping's resolver returns a null we get a NullPointerException:

22/05/24 08:12:12 980 ERROR DAGSchedulerEventProcessLoop: DAGSchedulerEventProcessLoop failed; shutting down SparkContext
java.lang.NullPointerException
	at scala.collection.convert.Wrappers$JListWrapper.isEmpty(Wrappers.scala:87)
	at org.apache.spark.deploy.yarn.SparkRackResolver.coreResolve(SparkRackResolver.scala:76)