[SPARK-44305] Broadcast operation is not required when no parameters are specified

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: 7mming7
• Created: 05/Jul/23 03:48
• Updated: 03/Aug/24 00:32

Description

The ability introduced by SPARK-14912, we can broadcast the parameters of the data source to the read and write operations, but if the user does not specify a specific parameter, the propagation operation will also be performed, which affects the performance has a greater impact, so we need to avoid broadcasting the full Hadoop parameters when the user does not specify a specific parameter

 

Issue Links

• duplicates: SPARK-44304 Broadcast operation is not required when no parameters are specified
• is duplicated by: SPARK-44304 Broadcast operation is not required when no parameters are specified
• relates to: SPARK-14912 Propagate data source options to Hadoop configurations
• links to: [Github] Pull Request #42037 (7mming7)
• links to: GitHub Pull Request #42037
• links to: GitHub Pull Request #46162