[SPARK-23630] Spark-on-YARN missing user customizations of hadoop config

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: Spark Core, YARN
• Target Version/s: 2.3.1
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 08/Mar/18 19:16
• Updated: 17/May/20 18:13
• Resolved: 09/Mar/18 18:37

Description

In my change to fix SPARK-22372, I removed some code that allowed user customizations of hadoop configs to take effect. Notably, these lines in YarnSparkHadoopUtil:

 override def newConfiguration(conf: SparkConf): Configuration = {
   val hadoopConf = new YarnConfiguration(super.newConfiguration(conf))
   hadoopConf.addResource(Client.SPARK_HADOOP_CONF_FILE)
   hadoopConf
 }

The addResource call is now missing so user configs are ignored (only the NM config is used).

Issue Links

• links to: [Github] Pull Request #20776 (vanzin)