[SPARK-21678] Disabling quotes while writing a dataframe

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.2.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Taran Saini
• Created: 09/Aug/17 13:24
• Updated: 10/Aug/17 00:42
• Resolved: 09/Aug/17 15:19

Description

Hi,

I have the my dataframe cloumn values which can contain commas, double quotes etc.
I am transforming the dataframes in order to ensure that all the required values are escaped.

However, on doing df.write.format("csv")
It again wraps the values in double quotes. How do I disable the same?
And even if the double quotes are there to stay why does it do the following :

L"\, p' Y a\, C G

is written as

"L\"\\, p' Y a\\, C G\\, H"

i.e double escapes the next already escaped values.
and if i myself escape like :

L\"\, p' Y a\, C G

then that is written as

 "L\\"\\, p' Y a\\, C G\\, H"

How do we just disable this automatic escaping of characters?

Issue Links

• is related to: SPARK-21684 df.write double escaping all the already escaped characters except the first one