[SPARK-24959] Do not invoke the CSV/JSON parser for empty schema

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.1
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Max Gekk
• Created: 28/Jul/18 16:06
• Updated: 12/Dec/22 18:11
• Resolved: 25/May/21 01:39

Description

Currently JSON and CSV parsers are called even if required schema is empty. Invoking the parser per each line has some non-zero overhead. The action can be skipped. Such optimization should speed up count(), for example.

Issue Links

• causes: SPARK-26745 Non-parsing Dataset.count() optimization causes inconsistent results for JSON inputs with empty lines
• links to: [Github] Pull Request #21909 (MaxGekk)
• links to: [Github] Pull Request #22920 (MaxGekk)
• links to: [Github] Pull Request #23667 (HyukjinKwon)
• links to: [Github] Pull Request #23708 (HyukjinKwon)
• links to: GitHub Pull Request #21909