[SPARK-6591] Python data source load options should auto convert common types into strings

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.2, 1.5.0
• Component/s: PySpark, SQL
• Labels: DataFrame DataSource
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Yijie Shen
• Reporter: Reynold Xin
• Shepherd: Davies Liu
• Created: 29/Mar/15 03:31
• Updated: 06/Aug/15 00:29
• Resolved: 06/Aug/15 00:29

Description

See the discussion at : https://github.com/databricks/spark-csv/pull/39

If the caller invokes

sqlContext.load("com.databricks.spark.csv", path = "cars.csv", header = True)

We should automatically turn header into "true" in string form.

We should do this for booleans and numeric values.

cc yhuai

Issue Links

• links to: [Github] Pull Request #7926 (yjshen)