[SPARK-17916] CSV data source treats empty string as null no matter what nullValue option is

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.1
• Fix Version/s: 2.4.0
• Component/s: SQL
• Target Version/s: 2.4.0
• Assignee: Maxim Gekk
• Reporter: Hossein Falaki
• Created: 13/Oct/16 18:17
• Updated: 11/Sep/18 06:46
• Resolved: 14/May/18 02:02

Description

When user configures nullValue in CSV data source, in addition to those values, all empty string values are also converted to null.

data:
col1,col2
1,"-"
2,""

spark.read.format("csv").option("nullValue", "-")

We will find a null in both rows.

Issue Links

• is duplicated by: SPARK-21768 spark.csv.read Empty String Parsed as NULL when nullValue is Set
• links to: [Github] Pull Request #20068 (aa8y)
• links to: [Github] Pull Request #21273 (MaxGekk)
• links to: [Github] Pull Request #22312 (koertkuipers)
• links to: [Github] Pull Request #22367 (MaxGekk)
• links to: [Github] Pull Request #22367 (MaxGekk)
• links to: [Github] Pull Request #22389 (MaxGekk)