[SPARK-44479] Support Python UDTFs with empty schema

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: PySpark
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 18/Jul/23 21:21
• Updated: 27/Jul/23 20:25
• Resolved: 27/Jul/23 00:56

Description

Support UDTFs with empty schema, for example:

>>> class TestUDTF:
...   def eval(self):
...     yield tuple()

Currently it fails with `useArrow=True`:

>>> udtf(TestUDTF, returnType=StructType())().collect()
Traceback (most recent call last):
...
ValueError: not enough values to unpack (expected 2, got 0)

whereas without Arrow:

>>> udtf(TestUDTF, returnType=StructType(), useArrow=False)().collect()
[Row()]

Otherwise, we should raise an error without Arrow, too, to be consistent.

Issue Links

• links to: [Github] Pull Request #42161 (ueshin)