[SPARK-40640] SparkHadoopUtil to set origin of hadoop/hive config options

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 03/Oct/22 16:24
• Updated: 13/Oct/22 19:28
• Resolved: 13/Oct/22 19:28

Description

Hadoop's Configuration class allows the caller to declare the origin of an option in the set(key, value, source) method.

Having spark do this for all programmatically set values helps people trying to debug where values came from. This is particularly relevant when looking at AWS env var propagation.

Issue Links

• links to: [Github] Pull Request #38084 (steveloughran)
• links to: [Github] Pull Request #38084 (steveloughran)