[SPARK-27589] Spark file source V2

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Gengliang Wang
• Created: 29/Apr/19 05:21
• Updated: 31/Jul/20 05:37

Description

Re-implement file sources with data source V2 API

Sub-Tasks

• Implement `doCanonicalize` in BatchScanExec for comparing query plan results: Resolved
• File source V2: return correct result for Dataset.inputFiles(): Resolved
• File source V2: support refreshing metadata cache: Resolved
• Revise the exception message of schema inference failure in file source V2: Resolved
• File source V2 table provider should be compatible with V1 provider: Resolved
• Support UDF input_file_name in file source V2: Resolved
• Support schema pruning in Orc V2: Resolved
• Migrate Parquet to File Data Source V2: Resolved
• File source V2: Invalidate cache data on overwrite/append: Resolved
• File source V2: Prune unnecessary partition columns: Resolved
• File source V2: return actual schema in method `FileScan.readSchema`: Resolved
• Fall back all v2 file sources in `InsertIntoTable` to V1 FileFormat: Resolved
• File source V2: Ignore empty files in load: Resolved
• Handles exceptions on proceeding to next record in FilePartitionReader: Resolved
• Migrate Text to File Data Source V2: Resolved
• File source v2 should validate data schema only: Resolved
• Improve file source V2 framework: Resolved
• Migrate JSON to File Data Source V2: Resolved
• Migrate CSV to File Data Source V2: Resolved
• Remove data source option check_files_exist: Resolved
• Support handling partition values in the abstraction of file source V2: Resolved
• File Source V2: avoid creating unnecessary FileIndex in the write path: Resolved
• Support schema validation in File Source V2: Resolved
• File source V2 write: create framework and migrate ORC to it: Resolved
• Allow OrcColumnarBatchReader to return less partition columns: Resolved
• Create file source V2 framework and migrate ORC read path: Resolved
• File source V2: support reporting statistics: Resolved
• Redact treeString of FileTable and DataSourceV2ScanExecBase: Resolved
• Allow altering table add columns with CSVFileFormat/JsonFileFormat provider: Resolved
• File source v2: support reading output of file streaming Sink: Resolved
• useV1SourceList configuration should be for all data sources: Resolved
• Migrate Avro to File source V2: Resolved
• Add PathCatalog for data source V2: Resolved
• File source V2: support partition pruning: Resolved
• Disable all the V2 file sources in Spark 3.0 by default: Resolved
• File source V2: Support partition pruning with subqueries: Open
• Add V1/V2 tests for TextSuite and WholeTextFileSuite: Resolved