[SPARK-40460] Streaming metrics is zero when select _metadata

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.3.1, 3.3.2, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: Structured Streaming
• Assignee: Yaohua Zhao
• Reporter: Yaohua Zhao
• Created: 16/Sep/22 00:34
• Updated: 28/Sep/22 03:34
• Resolved: 19/Sep/22 05:34

Description

Streaming metrics report all 0 (`processedRowsPerSecond`, etc) when selecting `_metadata` column. Because the logical plan from the batch and the actual planned logical are mismatched: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala#L348

Issue Links

• links to: [Github] Pull Request #37905 (Yaohua628)
• links to: [Github] Pull Request #37932 (Yaohua628)