[SPARK-25390] Data source V2 API refactoring

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Target Version/s: 3.0.0
• Assignee: Wenchen Fan
• Reporter: Wenchen Fan
• Created: 10/Sep/18 05:35
• Updated: 12/Dec/22 18:10
• Resolved: 09/Oct/19 05:26

Description

Currently it's not very clear how we should abstract data source v2 API. The abstraction should be unified between batch and streaming, or similar but have a well-defined difference between batch and streaming. And the abstraction should also include catalog/table.

An example of the abstraction:

batch: catalog -> table -> scan
streaming: catalog -> table -> stream -> scan

We should refactor the data source v2 API according to the abstraction

Issue Links

• blocks: SPARK-25186 Stabilize Data Source V2 API

Sub-Tasks

• data source V2 API refactoring (batch read): Resolved
• data source v2 API refactor (batch write): Resolved
• Remove SaveMode from data source v2 API: Resolved
• data source V2 API refactoring (micro-batch read): Resolved
• data source V2 API refactoring (continuous read): Resolved
• data source v2 API refactor: streaming write: Resolved
• remove streaming output mode from data source v2 APIs: Resolved
• create StreamingWrite at the begining of streaming execution: Resolved
• data source v2 API improvement: Resolved
• Add DataSourceV2 capabilities to check support for batch append, overwrite, truncate during analysis.: Resolved
• Add DataSourceV2 capabilities for streaming: Resolved
• move data source v2 API to catalyst module: Resolved
• table capabilty to skip the output column resolution: Resolved