[SPARK-45815] Provide an interface for Streaming sources to add _metadata columns

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.1
• Fix Version/s: 4.0.0
• Component/s: SQL, Structured Streaming
• Labels: pull-request-available
• Assignee: Yaohua Zhao
• Reporter: Yaohua Zhao
• Created: 07/Nov/23 02:45
• Updated: 09/Nov/23 11:36
• Resolved: 09/Nov/23 11:36

Description

Currently, only the native V1 file-based streaming source can read the `_metadata` column: https://github.com/apache/spark/blob/370870b7a0303e4a2c4b3dea1b479b4fcbc93f8d/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingRelation.scala#L63

 

Our goal is to create an interface that allows other streaming sources to add `_metadata` columns. For instance, we would like the Delta Streaming source, which you can find here: https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/sources/DeltaDataSource.scala#L49, to extend this interface and provide the `_metadata` column for its underlying storage format, such as Parquet.

Issue Links

• links to: GitHub Pull Request #43692