[NIFI-8377] CSVReader: quoting and trimming with value separator inconsistency

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.1, 1.13.2
• Component/s: Extensions
• Labels: CSVReader inconsistency parsing
• Assignee: Unassigned
• Reporter: Kasonnara
• Created: 29/Mar/21 16:58
• Updated: 10/Oct/21 00:08

Description

There is a little inconsistency of quoting and trimming when the value separator is present in the data and using Apache Common CSV parser.

Example:

case, A, B
quoted value,"aa",
quoted and trimmed value, "aa" ,
quoted value with comma,"a,a",
trimmed but wrongly unquoted value with comma, "a,a" ,

here in the 3 first cases, the value is correctly parsed

A : "aa", B : null

A : "aa", B : null

A : "a,a", B : null

so using separately quoting containing the value separator or spaces to trim works well.
 
However in the last example that combine quoted value separator and outer spaces to trim, then quoting fails

A : "\"a", B : "a\""

 
I think setting org.apache.commons.csv.CSVFormat.withIgnoreSurroundingSpaces(true) on the CSV parser would solve the issue, but I don't see the whole picture to tell if this would have other side effects.

Issue Links

• links to: GitHub Pull Request #4954