Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Won't Do
-
None
-
None
-
None
Description
In the 0.x baseline, templates were stored in a configurable directory, each in its own file. In 1.0, this was changed so that Templates would instead be stored in the flow.xml.gz directly. This was done for a couple of reasons:
- It made it easier to copy a flow.xml.gz from one node to another if needed and have all of the templates carried along with it
- It made it easier to include just a single byte[] when inheriting a flow and fingerprinting it
At that point in time, NiFi had experienced far less adoption than it has now, however, and the deployments that we envisioned were much smaller than what we see today. Now, it is not uncommon to see a deployment with many thousand processors and dozens or hundreds of templates. The cost of serializing these templates for every single change to the flow is more expensive than was realized at the time.
On a bare-metal server, with several dozen templates, I can sometimes see the saving of the flow.xml.gz take several (10-ish) seconds. This causes the UI to become sluggish at times and can cause cluster instability (though the save happens in the background, it does done in a lock, which can block other requests).
As a result, I believe we should consider reverting back to the old behavior of saving templates into individual files. This would result in better UI performance and better cluster stability.