[SPARK-50446] Concurrent level in Arrow-optimized Python UDF

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 28/Nov/24 06:37
• Updated: 03/Dec/24 01:49
• Resolved: 29/Nov/24 04:40

Description

For Arrow-optimized Python UDF, we can easily support concurrent execution in the UDF - it can speed up the case when individual UDF execution uses I/O intensively.

Issue Links

• links to: GitHub Pull Request #49000