[SPARK-41248] Add config flag to control before of JSON partial results parsing in SPARK-40646

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Ivan Sadikov
• Reporter: Ivan Sadikov
• Created: 24/Nov/22 05:06
• Updated: 04/Sep/23 21:24
• Resolved: 14/Dec/22 05:31

Description

This is a follow-up for https://issues.apache.org/jira/browse/SPARK-40646.

 

It was observed in internal benchmarks that the JSON partial results parsing can be 30% slower compared to parsing without the patch. I could not find a regression and the Apache Spark JSON benchmark results are very similar with and without SPARK-40646.

However, I would still like to add a config flag to enable/disable the feature in the case the regression is observed in users' queries.

Benchmark results are attached below.

 

Issue Links

• relates to: SPARK-40646 Fix returning partial results in JSON data source and JSON functions
• links to: [Github] Pull Request #38784 (sadikovi)