Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8681

Allow users to configure FlowFile Concurrency more specifically on ProcessGroup

    XMLWordPrintableJSON

    Details

      Description

      With NIFI-7476, users gained the ability to allow a single FlowFile to pass through a ProcessGroup before the next one is allowed through. I propose that we expand this concept to allow for grouping via FlowFile attribute. That is that there would be some hardcoded FlowFile attribute that incoming FlowFile's would be grouped via, and then only a single FlowFile from a grouping is allowed to be processed by the ProcessGroup at a time. This would essentially work as if there were multiple input queues that the ProcessGroup is pulling from - one for each unique FlowFile attribute value - and a FlowFile can only be pulled off the queue if there is no file from that group currently being processed.

      It is often the case that one would want to equitably share resources across some grouping, to prevent one grouping from starving out the others. This suggestion would provide an easy solution to such problems as setting the correct grouping would allow for every group to get the same resources without things getting starved out by one group having sent all of its FlowFiles first.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stewarta Alexander Stewart
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: