Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
Description
In the job topology, if the user specifies the concurrency of the operator, AdaptiveScheduler should support the operator's maximum parallelism equal to the user-specified parallelism during the scheduling process. And the minimum parallelism is equal to the number of slots available to the cluster.
This is especially useful in certain scenarios,
For example, the parallelism of an operator that consumes Kafka is specified to be equal to the number of partitions. Or you want to control the write rate of the operator, etc.