[SPARK-25493] CRLF Line Separators don't work in multiline CSVs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Justin Uang
• Reporter: Justin Uang
• Created: 20/Sep/18 20:53
• Updated: 12/Dec/22 18:10
• Resolved: 19/Oct/18 03:15

Description

CSVs with windows style crlf (carriage return line feed) don't work in multiline mode. They work fine in single line mode because the line separation is done by Hadoop, which can handle all the different types of line separators. In multiline mode, the Univocity parser is used to also handle splitting of records.

Issue Links

• is duplicated by: SPARK-26040 CSV Row delimiters not consistent between platforms
• is duplicated by: SPARK-25506 Spark CSV multiline with CRLF
• links to: [Github] Pull Request #22503 (justinuang)
• links to: [Github] Pull Request #22680 (justinuang)
• links to: GitHub Pull Request #22503