[SPARK-44876] Enable and fix test_parity_arrow_python_udf

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: PySpark
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 18/Aug/23 17:55
• Updated: 21/Aug/23 00:20
• Resolved: 21/Aug/23 00:20

Description

pyspark.sql.tests.connect.test_parity_arrow_python_udf is not listed in dev/sparktestsupport/modules.py, and it fails when running manually.

======================================================================
ERROR [0.072s]: test_register (pyspark.sql.tests.connect.test_parity_arrow_python_udf.ArrowPythonUDFParityTests)
----------------------------------------------------------------------
Traceback (most recent call last):
...
pyspark.errors.exceptions.base.PySparkRuntimeError: [SCHEMA_MISMATCH_FOR_PANDAS_UDF] Result vector from pandas_udf was not the required length: expected 1, got 38.