[SPARK-15689] Data source API v2

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Labels: SPIP releasenotes
• Target Version/s: 2.3.0
• Assignee: Wenchen Fan
• Reporter: Reynold Xin
• Shepherd: Reynold Xin
• Created: 01/Jun/16 05:32
• Updated: 09/Oct/19 15:35
• Resolved: 23/Jan/18 15:26

Description

This ticket tracks progress in creating the v2 of data source API. This new API should focus on:

1. Have a small surface so it is easy to freeze and maintain compatibility for a long time. Ideally, this API should survive architectural rewrites and user-facing API revamps of Spark.

2. Have a well-defined column batch interface for high performance. Convenience methods should exist to convert row-oriented formats into column batches for data source developers.

3. Still support filter push down, similar to the existing API.

4. Nice-to-have: support additional common operators, including limit and sampling.

Note that both 1 and 2 are problems that the current data source API (v1) suffers. The current data source API has a wide surface with dependency on DataFrame/SQLContext, making the data source API compatibility depending on the upper level API. The current data source API is also only row oriented and has to go through an expensive external data type conversion to internal data type.

Issue Links

• is related to: SPARK-15687 Columnar execution engine
• relates to: SPARK-19351 Support for obtaining file splits from underlying InputFormat
• links to: [Github] Pull Request #19136 (cloud-fan)
• links to: SPIP: Data Source API V2

Sub-Tasks

• data source v2 write path: Resolved
• clarify exception behaviors for all data source v2 interfaces: Resolved
• push down operators to data source before planning: Resolved
• mix-in interface should extend the interface it aimed to mix in: Resolved
• remove V2 from the class name of data source reader/writer: Resolved
• propagate session configs to data source read/write options: Resolved
• partitioning reporting: Resolved
• columnar reader interface: Resolved
• rename some APIs and classes to make their meaning clearer: Resolved
• DataSourceV2Options should have getInt, getBoolean, etc.: Resolved
• Rename ReadTask to DataReaderFactory: Resolved