[NIFI-6972] Stateless NiFi should support parallelism

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10.0
• Component/s: NiFi Stateless
• Labels: features
• Assignee: Unassigned
• Reporter: Adam Trump
• Created: 29/Dec/19 14:34
• Updated: 15/Mar/21 10:15

Description

Today, when running NiFi in stateless mode, it will run all flow files serially - meaning that only after a flow file finish running the whole flow, it will re run the flow for the next flow file.

In other words, it runs only one flow file at a time. Is there any reason it was implemented that way?

Instead, I think that there should be a new parameter when running the NiFi, which would be the maximum parallelism. In my opinion, it could be implemented quite easily:

Under StatelessFlow.java, in the run(final Queue<InMemoryFlowFile> output) method, in line 322, it should create a new thread each iteration, with the maximum being the given parameter.

This will allow the flow to run multiple flow file in parallel, and should be simple enough without breaking anything else...