[SPARK-15856] Revert API breaking changes made in SQLContext.range

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Assignee: Wenchen Fan
• Reporter: Cheng Lian
• Created: 10/Jun/16 00:28
• Updated: 11/Jun/16 22:28
• Resolved: 11/Jun/16 22:28

Description

In Spark 2.0, after unifying Datasets and DataFrames, we made two API breaking changes:

1. DataFrameReader.text() now returns Dataset[String] instead of DataFrame
2. SQLContext.range() now returns Dataset[java.lang.Long] instead of DataFrame

However, these two changes introduced several inconsistencies and problems:

1. spark.read.text() silently discards partitioned columns when reading a partitioned table in text format since Dataset[String] only contains a single field. Users have to use spark.read.format("text").load() to workaround this, which is pretty confusing and error-prone.
2. All data source shortcut methods in `DataFrameReader` return DataFrame (aka Dataset[Row]) except for DataFrameReader.text().
3. When applying typed operations over Datasets returned by spark.range(), weird schema changes may happen. Please refer to SPARK-15632 for more details.

Due to these reasons, we decided to revert these two changes.

Issue Links

• links to: [Github] Pull Request #13604 (cloud-fan)
• links to: [Github] Pull Request #13605 (cloud-fan)