[SPARK-44640] Improve error messages for Python UDTF returning non iterable

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: PySpark
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 02/Aug/23 17:59
• Updated: 07/Sep/23 01:43
• Resolved: 04/Aug/23 01:24

Description

When the return type of a UDTF is not an iterable, the error message can be confusing to users. For example for this UDTF:

@udtf(returnType="x: int")
class TestUDTF:
    def eval(self, a):
        return a 

Currently it fails with this error for regular UDTFs:

    return tuple(map(verify_and_convert_result, res))
TypeError: 'int' object is not iterable

And this error for arrow-optimized UDTFs:

    raise ValueError("DataFrame constructor not properly called!")
ValueError: DataFrame constructor not properly called!

Issue Links

• links to: [Github] Pull Request #42726 (allisonwang-db)