[STORM-3777] Ability to have a default doubling of resource when worker count increases

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: storm-core
• Assignee: Unassigned
• Reporter: Anton Alfred
• Created: 07/Jun/21 03:02
• Updated: 07/Jun/21 03:02

Description

Let's say we have a topology with the following executors configurations

reader: 1
bolt1: 2
bolt2: 4
bolt3: 4
Topology Memory: 1 GB
noOfWorkers: 1

When we submit the topology the bolts and readers are provided this many executors but when we change the noOfWorkers to say 2. The topology still is keeping the executors the same for the bolts though there is a second worker process.

Is it possible to have a default behavior or a policy configuration like an autoscale of API. If 1 Worker is present it would take this but when more than 1 worker is present it would double these by default (or a policy) without expecting to be provided manually again.

This way the configuration remains same and only the noOfWorkers keep changing for scale out and scale in.

This is related to the previous issue of simplified Auto Scaling Auto-Scaling Resources in a Topology

If this is present, a signal to increase or decrease the workers can be provided with lessor configuration changes.