[SPARK-45930] Allow non-deterministic Python UDFs in MapInPandas/MapInArrow

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Allison Wang
• Reporter: Allison Wang
• Created: 15/Nov/23 04:04
• Updated: 16/Nov/23 02:50
• Resolved: 16/Nov/23 02:50

Description

Currently if a Python udf is non-deterministic, the analyzer will fail with this error:[INVALID_NON_DETERMINISTIC_EXPRESSIONS] The operator expects a deterministic expression, but the actual expression is "pyUDF()", "a". SQLSTATE: 42K0E;

Issue Links

• links to: GitHub Pull Request #43810