[SPARK-40646] Fix returning partial results in JSON data source and JSON functions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Ivan Sadikov
• Reporter: Ivan Sadikov
• Created: 04/Oct/22 04:01
• Updated: 04/Sep/23 21:24
• Resolved: 17/Oct/22 08:47

Description

I recently found an issue when parsing the following JSON file:

{"a": {"x": 1, "y": true}, "b": {"x": 1}}
{"a": {"x": 2}, "b": {"x": 2}}

Trying to read such table with fixed schema where y is a struct column and not a boolean: 

val df = spark.read
  .schema("a struct<x: int, y: struct<x: int>>, b struct<x: int>")
  .json("path") 

results in the following answer:

a	                 b
null	                 null
{"x":2,"y":null}	{"x":2} 

Column b is valid and should be still parsed despite a having the wrong value.

 

This could be considered a follow-up to https://issues.apache.org/jira/browse/SPARK-33134.

Issue Links

• is related to: SPARK-44940 Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled
• is related to: SPARK-41248 Add config flag to control before of JSON partial results parsing in SPARK-40646
• relates to: SPARK-33134 Incorrect nested complex JSON fields raise an exception
• links to: [Github] Pull Request #38090 (sadikovi)
• links to: [Github] Pull Request #38090 (sadikovi)
• links to: [Github] Pull Request #38784 (sadikovi)
• links to: [Github] Pull Request #38784 (sadikovi)