[SPARK-46820] Fix error message regression by restoring new_msg

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Haejoon Lee
• Reporter: Haejoon Lee
• Created: 24/Jan/24 00:45
• Updated: 20/Feb/24 02:06
• Resolved: 20/Feb/24 02:06

Description

>>> from pyspark.sql.types import StructType, StructField, StringType, IntegerType
>>> schema = StructType([
...     StructField("name", StringType(), nullable=True),
...     StructField("age", IntegerType(), nullable=False)
... ])
>>> df = spark.createDataFrame([("asd", None])], schema)
pyspark.errors.exceptions.base.PySparkValueError: [CANNOT_BE_NONE] Argument `obj` cannot be None.

 

Issue Links

• links to: GitHub Pull Request #44859