[SPARK-14912] Propagate data source options to Hadoop configurations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 26/Apr/16 05:07
• Updated: 05/Jul/23 03:50
• Resolved: 26/Apr/16 17:59

Description

We currently have no way for users to propagate options to the underlying library that rely in Hadoop configurations to work. For example, there are various options in parquet-mr that users might want to set, but the data source API does not expose a per-job way to set it.

This patch propagates the user-specified options also into Hadoop Configuration.

Issue Links

• is duplicated by: SPARK-13912 spark.hadoop.* configurations are not applied for Parquet Data Frame Readers
• is related to: SPARK-44305 Broadcast operation is not required when no parameters are specified
• links to: [Github] Pull Request #12688 (rxin)