[SPARK-47970] Revisit skipped parity tests for PySpark Connect

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect, PySpark
• Assignee: Unassigned
• Reporter: Ruifeng Zheng
• Created: 24/Apr/24 06:48
• Updated: 25/Apr/24 08:07

Issue Links

• is related to: SPARK-44111 Prepare Apache Spark 4.0.0

Sub-Tasks

• Enable `QuietTest` in parity tests: Resolved
• Reuse `test_help_command` in Connect: Resolved
• Enable PandasUDFGroupedAggParityTests. test_grouped_with_empty_partition: Resolved
• Enable `ArrowParityTests.test_createDataFrame_empty_partition`: Resolved
• Enable `DefaultIndexParityTests. test_index_distributed_sequence_cleanup`: Resolved
• Enable `UDFParityTests.test_udf_timestamp_ntz`: Resolved
• Update the error class for `group.apply`: Resolved
• Enable PandasUDFScalarParityTests.{test_vectorized_udf_empty_partition, test_vectorized_udf_struct_with_empty_partition}: Resolved
• Enable `GroupedApplyInPandasTests. test_grouped_with_empty_partition`: Resolved
• `UserDefinedFunction.returnType` parse the DDL string: Resolved
• Enable `CogroupedApplyInPandasTests.test_wrong_args`: Resolved
• Enable `PandasUDFGroupedAggParityTests.test_manual`: Resolved
• Enable `ArrowPythonUDFParityTests.test_err_return_type`: Resolved
• Introduce a helper function to drop metadata: Resolved
• Fully enable PandasUDFParityTests. test_udf_wrong_arg: Resolved
• Implement the missing function validation in ApplyInXXX: Resolved