[SPARK-43039] Support custom fields in the file source _metadata column

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Ryan Johnson
• Reporter: Ryan Johnson
• Created: 05/Apr/23 15:36
• Updated: 12/Apr/23 03:03
• Resolved: 12/Apr/23 03:03

Description

Today, the schema of the file source _metadata column depends on the file format (e.g. parquet file format supports _metadata.row_index) but this is hard-wired into the FileFormat itself. Not only is this an ugly design, it also prevents custom file formats from adding their own fields to the _metadata column.

Issue Links

• links to: [Github] Pull Request #40677 (ryan-johnson-databricks)