Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.9.3, 0.9.6, 0.10.1, 1.0.1
-
None
Description
For legacy reasons, config TOPOLOGY-TASKS is considered first when schedule a topology, for a component, if user don’t specify TOPOLOGY-TASKS, storm just override it to be equal to component parallelism hint, and schedule based on TOPOLOGY-TASKS later on.
This works for the most cases, but not Rebalance command. Now, when do Rebalance, the StormBase :component->executors attribute will be overridden in Zookeeper which is used to partition component tasks into executors, as we said above, the TOPOLOGY-TASKS is considered here as the real tasks number for components, something goes weird here:
If we override a bigger executor numbers for a component when do rebalance, it just don’t work because smaller TOPOLOGY-TASKS [ not changed since first submitted at all ]is partitioned into bigger number of executors which read from ZooKeeper overridden by Rebalance command, but for smaller task, it works fine.
I see that storm support a command like this now: [storm rebalance topology-name [-w wait-time-secs] [-n new-num-workers] [-e component=parallelism]*] which indicate that user can override a component parallelism freely, i think it’s more sensible to support this and it's meaningless to have a restriction like before.
Attachments
Issue Links
- links to
