[DRILL-3178] csv reader should allow newlines inside quotes

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.9.0
• Component/s: Storage - Text & CSV

Environment

Ubuntu Trusty 14.04.2 LTS

• Assignee: F Méthot
• Reporter: Neal McBurnett
• Reviewer: Krystal
• Created: 23/May/15 04:44
• Updated: 03/Nov/16 18:15
• Resolved: 25/Oct/16 01:07

Description

When reading a csv file which contains newlines within quoted strings, e.g. via

select * from dfs.`/tmp/q.csv`;

Drill 1.0 says:

Error: SYSTEM ERROR: com.univocity.parsers.common.TextParsingException: Error processing input: Cannot use newline character within quoted string

But many tools produce csv files with newlines in quoted strings. Drill should be able to handle them.

Workaround: the csvquote program (https://github.com/dbro/csvquote) can encode embedded commas and newlines, and even decode them later if desired.

Issue Links

• is required by: DRILL-2785 Aggregate MAX query on a directory fails with SQLException
• links to: GitHub Pull Request #593