Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8681

Allow users to configure FlowFile Concurrency more specifically on ProcessGroup




      With NIFI-7476, users gained the ability to allow a single FlowFile to pass through a ProcessGroup before the next one is allowed through. I propose that we expand this concept to allow for grouping via FlowFile attribute. That is that there would be some hardcoded FlowFile attribute that incoming FlowFile's would be grouped via, and then only a single FlowFile from a grouping is allowed to be processed by the ProcessGroup at a time. This would essentially work as if there were multiple input queues that the ProcessGroup is pulling from - one for each unique FlowFile attribute value - and a FlowFile can only be pulled off the queue if there is no file from that group currently being processed.

      It is often the case that one would want to equitably share resources across some grouping, to prevent one grouping from starving out the others. This suggestion would provide an easy solution to such problems as setting the correct grouping would allow for every group to get the same resources without things getting starved out by one group having sent all of its FlowFiles first.


          Issue Links



              • Assignee:
                stewarta Alexander Stewart
              • Votes:
                0 Vote for this issue
                1 Start watching this issue


                • Created: