[SPARK-45551] Enhance PySpark testing utils

• Type: Umbrella
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: PySpark
• Assignee: Unassigned
• Reporter: Haejoon Lee
• Created: 16/Oct/23 11:26
• Updated: 14/Nov/23 05:08

Description

We have added several testing utils for PySpark testing: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.testing.assertDataFrameEqual.html.

We might need to enhance the functionality of those functions to make testing easier for users. The existing testing utils could be enhanced by:

• Adding new parameters to the existing utils
• Returning a debuggable object for failed assertions

and any other suggestions are welcome.

Sub-Tasks

• Introduce flexible parameters to assertDataFrameEqual: Resolved
• Deprecate assertPandasOnSparkEqual: Resolved
• Introduce flexible parameter to assertSchemaEqual: Resolved
• Returning a debuggable object for failed assertion: Resolved
• Introduce Pandas-like testing utils for Pandas API on Spark: Resolved