[SPARK-35376] Fallback config should override defaultValue

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: XiDuo You
• Created: 12/May/21 01:28
• Updated: 12/May/21 07:03

Description

Some config use `.fallbackConf` to reference another config but lost the defaultValue.

 

A negative case is `SQLConf.ADVISORY_PARTITION_SIZE_IN_BYTES.defaultValue` would return `None`. It could mislead user.

 

Issue Links

• links to: [Github] Pull Request #32510 (ulysses-you)