[SPARK-38323] Support the hidden file metadata in Streaming

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.0
• Component/s: SQL, Structured Streaming
• Assignee: Yaohua Zhao
• Reporter: Yaohua Zhao
• Created: 25/Feb/22 04:17
• Updated: 12/Dec/22 18:11
• Resolved: 01/Mar/22 13:10

Description

Currently, querying the hidden file metadata struct `_metadata` will fail with `readStream`, `writeStream` APIs.

spark
  .readStream
  ...
  .select("_metadata")
  .writeStream
  ...
  .start()

Need to expose the metadata output to `StreamingRelation` as well.

Issue Links

• links to: [Github] Pull Request #35676 (Yaohua628)