Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
When we lookup a component by ID, we do so by obtaining the Root Process Group and then calling findLocalConnectable(String id). This method obtains a read lock, then checks its map of Processors, its map of Input Ports, its map of Output Ports, and its map of Funnels. If no match is found, it then calls getRemoteProcessGroups() to iterate over each of those, looking for a Remote Input/Output Port with that ID. This call to getRemoteProcessGroups() creates a new HashSet that is then returned. If no match is found, we then call getProcessGroups() which also creates a new HashSet of ProcessGroup objects, and we iterate over those (recursively).
This means that for each call to lookup a component by ID, we have to create two HashSet}}s - for each Process Group on the canvas, until the component is found. Consider a flow that has a dozen Process Groups and several thousand Processors/ports/funnels. If we then click "Start" on the root group, we must create up to 24 {{HashSet objects and obtain 12 Read Locks. This is done for each component, so for 1,000 Processors it will create 24,000 {{HashSet}}s and obtain 12,000 Read Locks. Also, since this is a mutable request, this has to be done for both the first and second phase of the request, which results in a total of 48,000 {{HashSet}}s and 24,000 Read Locks being obtained.
Testing with 10,000 Processors I am seeing requests take well over 30 seconds to complete. All just to find a component by identifier. We can make this much more efficient.
Attachments
Issue Links
- links to
