[SPARK-14913] Simplify configuration API

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 26/Apr/16 06:48
• Updated: 10/Jan/17 05:34
• Resolved: 27/Apr/16 05:02

Description

We currently expose both Hadoop configuration and Spark SQL configuration in RuntimeConfig. I think we can remove the Hadoop configuration part, and simply generate Hadoop Configuration on the fly by passing all the SQL configurations into it. This way, there is a single interface (in Java/Scala/Python/SQL) for end-users.

Issue Links

• is duplicated by: SPARK-12307 ParquetFormat options should be exposed through the DataFrameReader/Writer options API
• relates to: SPARK-13912 spark.hadoop.* configurations are not applied for Parquet Data Frame Readers
• links to: [Github] Pull Request #12689 (rxin)