[SPARK-43967] Support Python UDTFs with empty return values

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: PySpark
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 05/Jun/23 05:55
• Updated: 18/Jul/23 06:00
• Resolved: 18/Jul/23 06:00

Description

Support UDTFs with empty returns, for example:

@udtf(returnType="a: int")
class TestUDTF:
    def eval(self, a: int):
        ... 

Currently, this will fail with the exception 

TypeError: 'NoneType' object is not iterable 

Another example

class TestUDTF:
    def eval(self, a: int):
        yield 

This will fail with the exceptions

java.lang.NullPointerException 

Note, arrow-optimized UDTFs already support this. This error only occurs with regular Python UDTFs.