[SPARK-42944] Support Python foreachBatch() in streaming spark connect

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Connect, Structured Streaming
• Labels: pull-request-available
• Epic Link: Streaming Spark Connect
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 28/Mar/23 04:23
• Updated: 08/Jun/24 01:54
• Resolved: 19/Jul/23 00:04

Description

Add support for foreachBatch() streaming spark connect. This might need deep dive into various complexities of arbitrary spark code since foreachBatch block. 

Issue Links

• links to: [Github] Pull Request #42675 (HyukjinKwon)
• links to: [Github] Pull Request #42675 (HyukjinKwon)
• links to: GitHub Pull Request #46853
• links to: GitHub Pull Request #46855