[SPARK-48330] Fix the python streaming data source timeout issue for large trigger interval

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SS
• Labels: pull-request-available
• Epic Link: unlabelled-SPARK-46866
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 18/May/24 19:41
• Updated: 20/May/24 22:57
• Resolved: 20/May/24 22:57

Description

Currently we run long running python worker process for python streaming source and sink to perform planning, commit and abort in driver side. Testing indicate that current implementation cause connection timeout error when streaming query has large trigger interval

For python streaming source, keep the long running worker archaetecture but set the socket timeout to be infinity to avoid timeout error.

For python streaming sink, since StreamingWrite is also created per microbatch in scala side, long running worker cannot be attached to s StreamingWrite instance. Therefore we abandon the long running worker architecture, simply call commit() or abort() and exit the worker and allow spark to reuse worker for us.

Issue Links

• links to: GitHub Pull Request #46651