[SPARK-24863] Report offset lag as a custom metrics for Kafka structured streaming source

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.0
• Component/s: Structured Streaming
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Arun Mahadevan
• Created: 19/Jul/18 20:19
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/21 01:44

Description

We can build on top of SPARK-24748 to report offset lag as a custom metrics for Kafka structured streaming source.

This is the difference between the latest offsets in Kafka the time the metrics is reported (just after a micro-batch completes) and the latest offset Spark has processed. It can be 0 (or close to 0) if spark keeps up with the rate at which messages are ingested into Kafka topics in steady state.

Issue Links

• relates to: SPARK-24933 SinkProgress should report written rows
• relates to: SPARK-25336 Revert SPARK-24863 and SPARK-24748
• links to: [Github] Pull Request #21819 (arunmahadevan)
• links to: [Github] Pull Request #22334 (zsxwing)
• links to: [Github] Pull Request #22334 (zsxwing)