[SPARK-13485] (Dataset-oriented) API evolution in Spark 2.0

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: SQL
• Labels: releasenotes
• Target Version/s: 2.0.0
• Assignee: Reynold Xin
• Reporter: Reynold Xin
• Created: 25/Feb/16 05:52
• Updated: 06/Jul/16 06:11
• Resolved: 30/Apr/16 07:22

Description

As part of Spark 2.0, we want to create a stable API foundation for Dataset to become the main user-facing API in Spark. This ticket tracks various tasks related to that.

The main high level changes are:

1. Merge Dataset/DataFrame
2. Create a more natural entry point for Dataset (SQLContext/HiveContext are not ideal because of the name "SQL"/"Hive", and "SparkContext" is not ideal because of its heavy dependency on RDDs)
3. First class support for sessions
4. First class support for some system catalog

See the design doc for more details.

Sub-Tasks

• Create SparkSession interface: Resolved
• Move SQLConf into an internal package: Resolved
• User-facing RuntimeConfig interface: Resolved
• User-facing catalog API: Resolved
• Design doc for configuration in Spark 2.0+: Closed
• Unify DataFrame and Dataset API: Resolved
• Improve user experience for typed aggregate functions in Scala: Resolved
• Improve user experience for typed aggregate functions in Java: Resolved
• Create a hivecontext-compatibility module: Resolved
• SparkSession should be case insensitive by default: Resolved
• Expose user-facing RuntimeConfig in SparkSession: Resolved
• Start SparkSession in REPL instead of SQLContext: Resolved
• Simplify configuration API: Resolved
• Python SparkSession API: Resolved
• Implement catalog and conf API in Python SparkSession: Resolved
• Simplify configuration API further: Resolved
• Use builder pattern to create SparkSession: Resolved
• Remove SparkSession.withHiveSupport: Resolved
• Use SparkSession in Scala/Python/Java example.: Resolved
• Use SparkSession instead of SQLContext in testsuites: Resolved
• Make SparkSession constructors private: Resolved
• Cleanup dependencies between SQLContext and SparkSession: Resolved
• Use builder pattern to create SparkSession in PySpark: Resolved
• Accept Dataset[_] in joins: Resolved
• RuntimeConfig.set should return Unit rather than RuntimeConfig itself: Resolved
• Remove experimental tag from DataFrameReader and DataFrameWriter: Resolved
• Remove experimental tag from Python DataFrame: Resolved
• Revert SPARK-14807 Create a hivecontext-compatibility module: Resolved
• SparkSession builder in python should also allow overriding confs of existing sessions: Resolved