[SPARK-35921] ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Build
• Assignee: Kousuke Saruta
• Reporter: Kousuke Saruta
• Created: 29/Jun/21 02:29
• Updated: 29/Jun/21 21:26
• Resolved: 29/Jun/21 21:25

Description

yarn sub-module contains config.properties.

spark.yarn.isHadoopProvided = ${spark.yarn.isHadoopProvided}

The ${spark.yarn.isHadoopProvided} part is replaced with true or false in build depending on whether Hadoop is provided or not (specified by -Phadoop-provided).
The edited config.properties will be loaded at runtime to control how to populate Hadoop-related classpath.

If we build with Maven, these process works but doesn't with SBT.

Issue Links

• links to: [Github] Pull Request #33121 (sarutak)