[SPARK-39404] Unable to query _metadata in streaming if getBatch returns multiple logical nodes in the DataFrame

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.3.2, 3.4.0
• Component/s: Structured Streaming
• Assignee: Yaohua Zhao
• Reporter: Yaohua Zhao
• Created: 07/Jun/22 18:11
• Updated: 22/Oct/22 10:27
• Resolved: 08/Jun/22 08:41

Description

Here: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala#L585

 

We should probably `transform` instead of `match`

Issue Links

• links to: [Github] Pull Request #36801 (Yaohua628)
• links to: [Github] Pull Request #38337 (Yaohua628)
• links to: [Github] Pull Request #38337 (Yaohua628)