[SPARK-13638] Support for saving with a quote mode

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: SQL
• Assignee: Jurriaan Pruis
• Reporter: Hyukjin Kwon
• Created: 03/Mar/16 05:45
• Updated: 12/Dec/22 18:11
• Resolved: 08/Jul/16 18:46

Description

https://github.com/databricks/spark-csv/pull/254

tobithiel reported this.

I'm dealing with some messy csv files and being able to just quote all fields is very useful,
so that other applications don't misunderstand the file because of some sketchy characters....

When writing there are several quote modes in apache commons csv. (See https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/QuoteMode.html)

This might have to be supported.

However, it looks univocity parser used for writing (it looks currently only this library is supported) does not support this quote mode. I think we can drop this backwards compatibility if we are not going to add apache commons csv.

This is a reminder that it might break backwards compatibility for the options, quoteMode.

Issue Links

• relates to: SPARK-15493 Allow setting the quoteEscapingEnabled flag when writing CSV
• links to: [Github] Pull Request #13374 (jurriaan)