Description
We currently have no way for users to propagate options to the underlying library that rely in Hadoop configurations to work. For example, there are various options in parquet-mr that users might want to set, but the data source API does not expose a per-job way to set it.
This patch propagates the user-specified options also into Hadoop Configuration.
Attachments
Issue Links
- is duplicated by
-
SPARK-13912 spark.hadoop.* configurations are not applied for Parquet Data Frame Readers
- Resolved
- is related to
-
SPARK-44305 Broadcast operation is not required when no parameters are specified
- Open
- links to