[SPARK-49536] Add error handling for python streaming data source record prefetching

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SS
• Labels: pull-request-available
• Assignee: Chaoqin Li
• Reporter: Chaoqin Li
• Created: 07/Sep/24 07:01
• Updated: 10/Sep/24 01:04
• Resolved: 10/Sep/24 01:04

Description

Currently there is an assert that return status code from the python worker is SpecialLengths.START_ARROW_STREAM when python source runner is prefetching records. To improve debugability, check the status code and rethrow an runtime error with detailed error message.

Issue Links

• links to: GitHub Pull Request #48023