[SPARK-43128] Streaming progress struct (especially in Scala)

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Connect, Structured Streaming
• Epic Link: Streaming Spark Connect
• Assignee: Yang Jie
• Reporter: Raghu Angadi
• Created: 13/Apr/23 17:45
• Updated: 17/May/23 07:27
• Resolved: 17/May/23 07:27

Description

Streaming spark connect transfers streaming progress as full “json”.

This works ok for Python since it does not have any schema defined.
But in Scala, it is a full fledged class. We need to decide if we want to match legacy Progress struct in spark-connect.

Issue Links

• links to: [Github] Pull Request #40892 (LuciferYang)
• links to: [Github] Pull Request #40892 (LuciferYang)
• links to: [Github] Pull Request #40895 (bogao007)