[SPARK-34297] Add metrics for data loss and offset out range for KafkaMicroBatchStream

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL, Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 30/Jan/21 07:53
• Updated: 23/Apr/21 20:57
• Resolved: 23/Apr/21 20:57

Description

When testing SS, I found it is hard to track data loss of SS reading from Kafka. The micro scan node has only one metric, number of output rows. Users have no idea how many times offsets to fetch are out of Kafak now, how many times data loss happens.

Issue Links

• links to: [Github] Pull Request #31398 (viirya)
• links to: [Github] Pull Request #31398 (viirya)
• links to: [Github] Pull Request #31944 (yijiacui-db)