[SPARK-34854] Report metrics for streaming source through progress reporter with Kafka source use-case

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Yijia Cui
• Reporter: Yijia Cui
• Created: 24/Mar/21 07:36
• Updated: 05/May/21 08:28
• Resolved: 05/May/21 08:28

Description

Report Metrics for streaming source through progress reporter to users.

Add kafka micro batch streaming use-case to report stats of # of offsets for the current offset falling behind the latest.

 

SPARK-34366 and SPARK-34297 report metrics in spark ui, but this issue reports metrics through progress report, making it available to users via listener.

Issue Links

• links to: [Github] Pull Request #31944 (yijiacui-db)
• links to: [Github] Pull Request #31944 (yijiacui-db)