[SPARK-24748] Support for reporting custom metrics via Streaming Query Progress

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Arun Mahadevan
• Created: 06/Jul/18 01:50
• Updated: 12/Dec/22 18:10
• Resolved: 08/Oct/19 05:42

Description

Currently the Structured Streaming sources and sinks does not have a way to report custom metrics. Providing an option to report custom metrics and making it available via Streaming Query progress can enable sources and sinks to report custom progress information (E.g. the lag metrics for Kafka source).

Issue Links

• is related to: SPARK-24933 SinkProgress should report written rows
• relates to: SPARK-24647 Sink Should Return Writen Offsets For ProgressReporting
• relates to: SPARK-25336 Revert SPARK-24863 and SPARK-24748
• links to: [Github] Pull Request #21721 (arunmahadevan)
• links to: [Github] Pull Request #22296 (HyukjinKwon)
• links to: [Github] Pull Request #22299 (arunmahadevan)
• links to: [Github] Pull Request #22334 (zsxwing)
• links to: [Github] Pull Request #22334 (zsxwing)

Sub-Tasks

• Report offset lag as a custom metrics for Kafka structured streaming source: Resolved