Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4836

Allow QueryDatabaseTables to send out batches of flow files while result set is being processed

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: Extensions
    • Labels:
      None

      Description

      Currently QueryDatabaseTable (QDT) will not transfer the outgoing flowfiles to the downstream relationship(s) until the entire result set has been processed (regardless of whether Max Rows Per Flow File is set). This is so the maxvalue.* and fragment.count attributes can be set correctly for each flow file.

      However for very large result sets, the initial fetch can take a long time, and depending on the setting of Max Rows Per FlowFile, there could be a great number of FlowFiles transferred downstream as a large burst at the end of QDT execution.

      It would be nice for the user to be able to choose to have FlowFiles be transferred downstream while the result set is still being processed. This alleviates the "large burst at the end" by replacing it with smaller output batches during processing. The tradeoff will be that if an Output Batch Size is set, then the maxvalue.* and fragment.count attributes will not be set on the outgoing flow files.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mattyb149 Matt Burgess
                Reporter:
                mattyb149 Matt Burgess
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: