[SPARK-25241] Configurable empty values when reading/writing CSV files

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.1, 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Mario Molina
• Reporter: Mario Molina
• Created: 25/Aug/18 17:38
• Updated: 12/Dec/22 18:11
• Resolved: 11/Sep/18 12:51

Description

There is an option in the CSV parser to set values when we have empty values in the CSV files or in our dataframes.

Currently, this option cannot be configured and always sets a default value (empty string for reading and `""` for writing).

I think it'd be interesting to enable this option in the CSV reader/writer to allow customizing these empty values.

Issue Links

• is duplicated by: SPARK-25739 Double quote coming in as empty value even when emptyValue set as null
• links to: [Github] Pull Request #22234 (mmolimar)
• links to: [Github] Pull Request #22367 (MaxGekk)
• links to: [Github] Pull Request #22389 (MaxGekk)
• links to: [Github] Pull Request #22389 (MaxGekk)