[STORM-3776] Ability to change the configuration of a topology without restarting

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Component/s: storm-core
• Assignee: Unassigned
• Reporter: Anton Alfred
• Created: 05/Jun/21 02:16
• Updated: 02/Jul/24 09:54
• Resolved: 02/Jul/24 09:54

Description

Have gone through this issue of Autoscale https://issues.apache.org/jira/browse/STORM-594.

We are trying a simpler approach for scaling in and out.

We create a topic in Kafka and would want to start processing the messages in the topology by having one worker.

As the lag in topic in Kafka increases, we want to be able to intimate Storm nimbus to change the topology workers and the number of executors in the bolts. Once this is done the topology should detect this change and scale out without needing a restart. Once we monitor the Kafka lag coming down, we want to send the scale in signal to the Storm nimbus to change back.

Currently due to no support for dynamic changes, we are forced to restart the topology when configuration changes...