[SPARK-31255] DataSourceV2: Add metadata columns

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 25/Mar/20 22:27
• Updated: 21/Dec/21 14:51
• Resolved: 18/Nov/20 22:23

Description

DSv2 should support reading additional metadata columns that are not in a table's schema. This allows users to project metadata like Kafka's offset, timestamp, and partition. It also allows other sources to expose metadata like file and row position.

Issue Links

• causes: SPARK-33524 Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform`
• links to: [Github] Pull Request #30415 (rdblue)
• links to: GitHub Pull Request #28027