[STORM-2823] Ability to have an option to combine topologies at run time in a single process space

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.1
• Component/s: storm-core
• Environment: CentOS 7, Docker
• Assignee: Unassigned
• Reporter: Anton Alfred
• Created: 19/Nov/17 04:39
• Updated: 20/Nov/17 03:14

Description

Unlike an API server which service multiple APIs within the same process space, the Topologies needs to run in separate processes.
Lets say we have Topology TP-1 which use 1 GB of memory.
Now we create the same for n Topologies of TP-1..... TP-n
As the topologies increase the memory allocation is now multiplied by the number of topologies.
This design though scalable is not similar to the API route we have before which was within the same process space.

So in a micros services world, each topology would be responsible for a similar set of objects, like employee, customer, product, order, order details etc.

As the number of topologies increase the worker allocation is not sufficient. Most topologies are not utilized fully but since these are in different process space the memory allocated can't be used.

If we have an ability to say that TP-1 — TP 10 Can run within the same process space but behave like individual topologies we could conserve the resource usage.

Now user are forced to combine topologies to the hardware provided with "if" logics to route the correct object that needs to be processed.

This way one can still configure topologies as API in the same API server and reuse resources collectively for related group of topologies acting as micro services.