[SPARK-43226] Define extractors for file-constant metadata columns

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Ryan Johnson
• Reporter: Ryan Johnson
• Created: 21/Apr/23 03:43
• Updated: 25/Apr/23 08:11
• Resolved: 25/Apr/23 08:11

Description

File-source constant metadata columns are often derived indirectly from file-level metadata values rather than exposing those values directly. For example, _metadata.file_name is currently hard-coded in FileFormat.updateMetadataInternalRow as:

 

UTF8String.fromString(filePath.getName)

 

We should add support for metadata extractors, functions that map from PartitionedFile to Literal, so that we can express such columns in a generic way instead of hard-coding them.

We can't just add them to the metadata map because then they have to be pre-computed even if it turns out the query does not select that field.