[SPARK-37273] Hidden File Metadata Support for Spark SQL

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Labels: release-notes
• Assignee: Yaohua Zhao
• Reporter: Yaohua Zhao
• Created: 11/Nov/21 00:54
• Updated: 12/Dec/22 18:10
• Resolved: 21/Dec/21 14:51

Description

Provide a new interface in Spark SQL that allows users to query the metadata of the input files for all file formats, expose them as built-in hidden columns meaning users can only see them when they explicitly reference them (e.g. file path, file name)

Issue Links

• relates to: SPARK-37767 Follow-up Improvements of Hidden File Metadata Support for Spark SQL
• links to: [Github] Pull Request #34575 (Yaohua628)