[NIFI-4836] Allow QueryDatabaseTables to send out batches of flow files while result set is being processed

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Extensions
• Assignee: Matt Burgess
• Reporter: Matt Burgess
• Created: 01/Feb/18 15:06
• Updated: 17/Feb/18 14:56
• Resolved: 17/Feb/18 14:56

Description

Currently QueryDatabaseTable (QDT) will not transfer the outgoing flowfiles to the downstream relationship(s) until the entire result set has been processed (regardless of whether Max Rows Per Flow File is set). This is so the maxvalue.* and fragment.count attributes can be set correctly for each flow file.

However for very large result sets, the initial fetch can take a long time, and depending on the setting of Max Rows Per FlowFile, there could be a great number of FlowFiles transferred downstream as a large burst at the end of QDT execution.

It would be nice for the user to be able to choose to have FlowFiles be transferred downstream while the result set is still being processed. This alleviates the "large burst at the end" by replacing it with smaller output batches during processing. The tradeoff will be that if an Output Batch Size is set, then the maxvalue.* and fragment.count attributes will not be set on the outgoing flow files.

Issue Links

• links to: GitHub Pull Request #2447