Description
To be able to accommodate for loading and running several different topologies on a cluster, one may want to assign more slots per node than there are CPUs available (many slots per CPU).
For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9 slots total) limits the number of task that can be loaded at any one time.
=> Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to load more topologies.
In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a parallelism of 9, 9 slots out of 27 are used, but the distribution strategy gives preference to slots on the same host as the stream source to limit network IOs: as a consequence, the first job using 9 slots has the 9 slots allocated on a single host (the one on the same host as the stream source), even though this host only has 3 CPUs available.
These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes (6 CPUs) stay idle.
The distribution strategy should take into account the number of CPUs available along with the number of slots rather than rely only on the slot availability.
Attachments
Issue Links
- duplicates
-
FLINK-1003 Spread out scheduling strategy
-
- Closed
-