[SPARK-25086] Incorrect Default Value For "escape" For CSV Files

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: David Wilcox
• Created: 10/Aug/18 14:18
• Updated: 12/Dec/22 18:11
• Resolved: 14/Aug/18 03:18

Description

The RFC for CSV files (https://tools.ietf.org/html/rfc4180) indicates that the way that a double-quote is escaped is by preceding it with another double-quote:

7. If double-quotes are used to enclose fields, then a double-quote appearing inside a field must be escaped by preceding it with another double quote. For example: "aaa","b""bb","ccc"

Your default value for "escape" violates the RFC. I think that we should fix the default value to be ", and those that want {{}} to escape can override for non-RFC-conforming CSV files.

Issue Links

• duplicates: SPARK-22236 CSV I/O: does not respect RFC 4180