[NIFI-1280] Create QueryFlowFile Processor

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 11/Dec/15 16:08
• Updated: 14/May/17 12:37
• Resolved: 11/Apr/17 23:34

Description

We should have a Processor that allows users to easily filter out specific columns from CSV data. For instance, a user would configure two different properties: "Columns of Interest" (a comma-separated list of column indexes) and "Filtering Strategy" (Keep Only These Columns, Remove Only These Columns).

We can do this today with ReplaceText, but it is far more difficult than it would be with this Processor, as the user has to use Regular Expressions, etc. with ReplaceText.

Eventually a Custom UI could even be built that allows a user to upload a Sample CSV and choose which columns from there, similar to the way that Excel works when importing CSV by dragging and selecting the desired columns? That would certainly be a larger undertaking and would not need to be done for an initial implementation.

Issue Links

• blocks: NIFI-3658 Add Processor capable of converting between different "record formats" with the same schema
• blocks: NIFI-3659 Add Processor capable of splitting up flowfiles that are in a "record format"
• is blocked by: NIFI-2039 Provide ability to get InputStream from FlowFile without using Process Session callback
• relates to: NIFI-3717 Add Schema Registry Controller Service for interacting with the Hortonworks Schema Registry
• relates to: NIFI-3682 Add "Schema Access Strategy" to Record Readers and Writers
• links to: GitHub Pull Request #420
• links to: GitHub Pull Request #1652