[NIFI-6050] Add FlattenRecord Processor

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Core Framework
• Labels: features
• Assignee: Unassigned
• Reporter: Adam Fisher
• Created: 18/Feb/19 00:32
• Updated: 18/Feb/19 00:43

Description

FlattenRecord

Tags:

record, recordpath, rpath, merge, group, content, correlation, stream, bin, organize, flatten

CapabilityDescription: 
Receives record-oriented data (i.e., data that can be read by the configured Record Reader) and evaluates one or more RecordPaths determined by user-defined properties against each record in the incoming FlowFile. Each record is then merged with other "like records" and the resulting FlowFile content will contain a single merged record with all of the fields from the group of records matching the RecordPaths. Two records are considered alike if they have the same value for all configured RecordPaths.

Array and plain record properties are merged recursively. Other complex types and value types are overwritten by assignment. FlowFile content is scanned from top to bottom. Subsequent records with matching field names overwrite field assignments of previous records in the same matching group. All other unique fields appearing in a matched group of records will be merged together.

See Additional Details on the Usage page for more information and examples.

Properties:

• Record Reader - Specifies the Controller Service to use for reading incoming data
• Record Writer - Specifies the Controller Service to use for writing out the records
• Scanning Strategy - Specifies how the FlowFile content is searched to find matching records to merge based on the user-defined RecordPath properties.
  • Global - Matching RecordPaths could appear anywhere in the file. This can be less performant than the Linear strategy for large files since a subset of matching records will be held in memory for each unique combination of RecordPaths before being written to the outgoing FlowFile.
  • Linear - Records with matching RecordPaths appear sequentially next to each other in the file. If the incoming FlowFile has this characteristic, it is better suited to processing large files. RecordPaths that match but are not sequentially next to each other in the FlowFile will be placed in separate matching groups and therefore be a separate record in the outgoing FlowFile content.

• Merge Strategy - Specifies the algorithm used to merge records.
  • Defragment - Combines fragments that are associated by RecordPaths back into a single record.
  • ... Need ideas for other merge strategies otherwise this property can go away.