Description
This is a follow-up for https://issues.apache.org/jira/browse/SPARK-40646.
It was observed in internal benchmarks that the JSON partial results parsing can be 30% slower compared to parsing without the patch. I could not find a regression and the Apache Spark JSON benchmark results are very similar with and without SPARK-40646.
However, I would still like to add a config flag to enable/disable the feature in the case the regression is observed in users' queries.
Benchmark results are attached below.
Attachments
Attachments
Issue Links
- relates to
-
SPARK-40646 Fix returning partial results in JSON data source and JSON functions
- Resolved
- links to