[NIFI-4122] GetMongo should be able to group results into a set of flowfiles

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: Extensions
• Labels: getmongo mongodb nifi
• Assignee: Unassigned
• Reporter: Mike Thomsen
• Created: 26/Jun/17 14:57
• Updated: 29/Jun/17 10:45
• Resolved: 29/Jun/17 10:45

Description

GetMongo should be able to take a user-defined limit and group results by that size into flowfiles rather than having only the ability to do a 1:1 relationship between result and flowfile.

For example, if the user specifies 100, 100 results should be grouped together and turned into a JSON array that can be broken up later as needed.

This need arose when doing a bulk data ingestion from Mongo. We had shy of 400k documents, and the 1:1 generation of flowfiles blew right through our limits on the content repository. Adding this feature would make it feasible to control that sort of behavior more thoroughly for events like bulk ingestion.

Issue Links

• links to: GitHub Pull Request #1948