[SPARK-21684] df.write double escaping all the already escaped characters except the first one

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Taran Saini
• Created: 09/Aug/17 18:29
• Updated: 21/May/19 04:12
• Resolved: 21/May/19 04:12

Description

Hi,

If we have a dataframe with the column value as

 ab\,cd\,ef\,gh 

Then while writing it is being written as

 "ab\,cd\\,ef\\,gh" 

i.e it double escapes all the already escaped commas/delimiters but not the first one.
This is weird behaviour considering either it should do for all or none.
If I do mention df.option("escape","") as empty then it solves this problem but the double quotes inside the same value if any are preceded by a special char i.e '\u00'. Why does it do so when the escape character is set as ""(empty)?

Issue Links

• relates to: SPARK-21678 Disabling quotes while writing a dataframe