[SPARK-25243] Use FailureSafeParser in from_json

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Max Gekk
• Reporter: Max Gekk
• Created: 26/Aug/18 15:57
• Updated: 15/Feb/21 21:25
• Resolved: 24/Oct/18 11:10

Description

The FailureSafeParser is used in parsing JSON, CSV files and dataset of strings. It supports the PERMISSIVE, DROPMALFORMED and FAILFAST modes. The ticket aims to make the from_json function compatible to regular parsing via FailureSafeParser and support above modes

Issue Links

• causes: SPARK-34441 from_json documentation is wrong about malformed JSONs output
• is duplicated by: SPARK-23194 from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls
• links to: [Github] Pull Request #22237 (MaxGekk)