[SPARK-33955] Add latest offsets to source progress

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 01/Jan/21 21:02
• Updated: 03/Jan/21 09:32
• Resolved: 03/Jan/21 09:32

Description

Currently we record start and end offsets per source in streaming process. Latest offset is an important information for streaming process but the progress lacks of this info. We should add latest offset to source progress.

Issue Links

• is related to: SPARK-33833 Allow Spark Structured Streaming report Kafka Lag through Burrow
• links to: [Github] Pull Request #30988 (viirya)