[SPARK-18579] spark-csv strips whitespace (pyspark)

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.2
• Fix Version/s: 2.2.0
• Component/s: Input/Output
• Target Version/s: 2.2.0
• Assignee: Hyukjin Kwon
• Reporter: Adrian Bridgett
• Created: 24/Nov/16 16:08
• Updated: 12/Dec/22 18:11
• Resolved: 23/Mar/17 07:25

Description

ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace are supported on CSV reader (and defaults to false).

However these are not supported options on the CSV writer and so the library defaults take place which strips the whitespace.

I think it would make the most sense if the writer semantics matched the reader (and did not alter the data) however this would be a change in behaviour. In any case it'd be great to have the option to strip or not.

Issue Links

• is duplicated by: SPARK-19942 DataFrameWriter - CSV options "ignoreLeadingWhiteSpace" and "ignoreTrailingWhiteSpace" Not working
• is duplicated by: SPARK-21442 Spark CSV writer trims trailing spaces
• links to: [Github] Pull Request #17310 (HyukjinKwon)