[SPARK-41597] Improve PySpark errors

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.4.0, 3.5.0
• Fix Version/s: 3.4.0
• Component/s: Connect, PySpark, Tests
• Assignee: Haejoon Lee
• Reporter: Haejoon Lee
• Created: 20/Dec/22 10:21
• Updated: 09/Feb/23 10:26
• Resolved: 09/Feb/23 10:26

Description

This ticket aims to introduce new PySpark framework to centralize the PySpark error message into single path and improve the error message more actionable and consistency.

This umbrella JIRA might includes:

• Introduce new error framework for PySpark
• Migrate existing errors generated by Python driver into error classes.
• Migrate existing errors generated by Python worker into error classes.
• Migrate existing errors generated by Py4J into error classes.
• Introduce test utils for testing errors by its error class instead of error messages.
• Improve the error messages.
• Documentation for PySpark error framework.

Sub-Tasks

• Introduce new PySpark package: pyspark.errors: Resolved
• Migrate the errors from `pyspark/sql/functions.py` into error class.: Resolved
• Migrate the Spark Connect errors into PySpark error framework.: Resolved
• Migrate errors thrown by JVM into PySpark Exception.: Resolved
• Add guideline for PySpark errors.: Resolved
• Introduce `PySparkValueError` and `PySparkTypeError`: Resolved
• Migrate the `TypeError` from `pyspark/sql/dataframe.py` into `PySparkTypeError`.: Resolved
• Migrate `TypeError` into error framework for Spark Connect functions: Resolved
• Add `SparkConnectIllegalArgumentException` to handle Spark Connect error precisely.: Resolved
• Refine error message by using Python types.: Resolved
• Add test for detecting duplicated error class: Resolved
• Update Debugging PySpark documents to show error message properly: Resolved
• Introduce base hierarchy to exceptions.: Resolved
• Migrate `TypeError` into error framework for Spark Connect column API.: Resolved
• Document error class list: Resolved
• Provide a clear error message for unsupported JVM attributes.: Resolved
• Introduce more basic exceptions.: Resolved