[SPARK-13485] (Dataset-oriented) API evolution in Spark 2.0

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Labels: releasenotes
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 25/Feb/16 05:52
• Updated: 06/Jul/16 06:11
• Resolved: 30/Apr/16 07:22

Description

As part of Spark 2.0, we want to create a stable API foundation for Dataset to become the main user-facing API in Spark. This ticket tracks various tasks related to that.

The main high level changes are:

1. Merge Dataset/DataFrame
2. Create a more natural entry point for Dataset (SQLContext/HiveContext are not ideal because of the name "SQL"/"Hive", and "SparkContext" is not ideal because of its heavy dependency on RDDs)
3. First class support for sessions
4. First class support for some system catalog

See the design doc for more details.