[SPARK-24045] Create base class for file data source v2

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Gengliang Wang
• Created: 22/Apr/18 13:47
• Updated: 29/Apr/19 05:50
• Resolved: 29/Apr/19 05:50

Description

From https://github.com/apache/spark/pull/20933, we can see that during File data source V2 migration,  falling back to V1 implementation is necessary in following cases1. File datasource V2 might be implemented partially during migration.
  E.g. if [[ReadSupport]] is implemented while [[WriteSupport]] is not,
  write path should fall back to V1 implementation.

2. File datasource V2 implementations cause regression.
3. Catalog support is required, which is still under development for data source V2.

 

Create a base class FileDataSourceV2 and resolve the falling back problems.

Issue Links

• links to: [Github] Pull Request #21123 (gengliangwang)
• links to: GitHub Pull Request #21123