[SPARK-41199] Streaming query metrics is broken with mixed-up usage of DSv1 streaming source and DSv2 streaming source

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.2, 3.3.1, 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 18/Nov/22 20:46
• Updated: 19/Nov/22 13:42
• Resolved: 19/Nov/22 13:42

Description

(It seems like a long standing issue. It probably applies to 2.x as well. I just marked the version line we did not EOL.)

If a streaming query contains both DSv1 and DSv2 streaming sources together, it only collects metrics properly for DSv1 sources.

 

Issue Links

• links to: [Github] Pull Request #38719 (HeartSaVioR)