[SPARK-35045] Add an internal option to control input buffer in univocity

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.0.3, 3.1.2, 3.2.0
• Component/s: SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 13/Apr/21 06:19
• Updated: 12/Dec/22 17:51
• Resolved: 13/Apr/21 12:12

Description

https://github.com/apache/spark/pull/31858 fixed it to respect Univocity's default buffer to:

• Firstly, it's best to trust their judgement on the default values. Also 128 is too low.
• Default values arguably have more test coverage in Univocity.
• It will also fix uniVocity/univocity-parsers#449 which is a regression compared to Spark 2.4

To prevent related side effect, we should have a workaround to change the buffer size.

Issue Links

• links to: [Github] Pull Request #32145 (HyukjinKwon)
• links to: [Github] Pull Request #32231 (HyukjinKwon)
• links to: [Github] Pull Request #32231 (HyukjinKwon)