[SPARK-13764] Parse modes in JSON data source

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 09/Mar/16 05:36
• Updated: 12/Dec/22 18:11
• Resolved: 21/Mar/16 07:43

Description

Currently, JSON data source just fails to read if some JSON documents are malformed.

PERMISSIVE : sets other fields to null when it meets a corrupted record and puts the malformed string into a new field configured by spark.sql.columnNameOfCorruptRecord. When a schema is set by user, it sets null for extra fields.
DROPMALFORMED : ignores the whole corrupted records.
FAILFAST: throws an exception when it meets corrupted records.

Issue Links

• relates to: SPARK-3308 Ability to read JSON Arrays as tables
• links to: [Github] Pull Request #11756 (HyukjinKwon)