[DRILL-7869] CSV files can't mix line breaks \x0d Vs. \x0d\x0a

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.19.0
• Component/s: Storage - Text & CSV
• Assignee: Unassigned
• Reporter: Curtis Lambert
• Created: 24/Feb/21 14:40
• Updated: 10/Jun/21 06:51

Description

Querying CSV files with \x0d new line delimiters results in "DATA_READ ERROR: Column exceeds maximum length of 1024" with the default configuration.

The \x0d new line isn't used to break lines resulting in the entire file being read in as a single record. This is configurable as "delimiter" in the format but if you have mixed csv files with different line endings it's problematic. If I have files with both \x0d and \x0d\x0a new lines (\r\n) and need to be able to read both without having to change the configuration between queries.