[SPARK-47273] Implement python stream writer interface

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SS
• Labels: pull-request-available
• Epic Link: unlabelled-SPARK-46866
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 04/Mar/24 23:20
• Updated: 22/Apr/24 05:52
• Resolved: 27/Mar/24 03:51

Description

In order to support developing spark streaming sink in python, we need to implement python stream writer interface.

Reuse PythonPartitionWriter to implement the serialization and execution of write callback in executor.

Implement python worker process to run python streaming data sink committer and communicate with JVM through socket in spark driver. For each python streaming data sink instance there will be a long live python worker process created. Inside the python process, the python write committer will receive abort or commit function call and send back result through socket.

Issue Links

• links to: GitHub Pull Request #45305