[NIFI-4496] Improve performance of CSVReader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Extensions
• Assignee: Matt Burgess
• Reporter: Matt Burgess
• Created: 18/Oct/17 14:33
• Updated: 22/Dec/17 13:57
• Resolved: 22/Dec/17 13:57

Description

During some throughput testing, it was noted that the CSVReader was not as fast as desired, processing less than 50k records per second. A look at this benchmark implies that the Apache Commons CSV parser (used by CSVReader) is quite slow compared to others.

From that benchmark it appears that CSVReader could be enhanced by using a different CSV parser under the hood. Perhaps Jackson is the best choice, as it is fast when values are quoted, and is a mature and maintained codebase.

Issue Links

• links to: GitHub Pull Request #2245