[SPARK-11319] PySpark silently accepts null values in non-nullable DataFrame fields.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 2.0.0
• Component/s: PySpark, SQL
• Assignee: Wenchen Fan
• Reporter: Kevin Cox
• Created: 26/Oct/15 16:53
• Updated: 06/May/16 21:18
• Resolved: 06/May/16 21:18

Description

Running the following code with a null value in a non-nullable column silently works. This makes the code incredibly hard to trust.

In [2]: from pyspark.sql.types import *
In [3]: sqlContext.createDataFrame([(None,)], StructType([StructField("a", TimestampType(), False)])).collect()
Out[3]: [Row(a=None)]

Issue Links

• duplicates: SPARK-13740 add null check for _verify_type in types.py
• is related to: SPARK-11868 wrong results returned from dataframe create from Rows without consistent schma on pyspark
• is related to: SPARK-13740 add null check for _verify_type in types.py
• links to: [Github] Pull Request #11785 (francoisprunier)