[SPARK-17763] JacksonParser silently parses null as 0 when the field is not nullable

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Hyukjin Kwon
• Created: 03/Oct/16 07:48
• Updated: 12/Dec/22 17:51
• Resolved: 21/May/19 04:15

Description

It seems JacksonParser parses the json when the field is not nullable, it parses some data as 0 silently. For example,

val testJson = """{"nullInt":null}""" :: Nil
val testSchema = StructType(StructField("nullInt", IntegerType, false) :: Nil)
val data = spark.sparkContext.parallelize(testJson)
spark.read.schema(testSchema).json(data).show()

shows

+-------+
|nullInt|
+-------+
|      0|
+-------+

This will also affect from_json function.

Issue Links

• is related to: SPARK-23173 from_json can produce nulls for fields which are marked as non-nullable
• links to: [Github] Pull Request #15329 (HyukjinKwon)