Details
-
Improvement
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
None
-
None
Description
I have a completed job, and each vertex may have different parallelism, and what troubles me is that the metric 'cpu used' differs among machines.
It comes to be good when I upgraded to use flink1.10, and add 'cluster.evenly-spread-out-slots: true' to flink config. This is good, while sometimes it is not enough.
For example, I have 5 taskmanagers(each deployed in one machine). I have a job, and some vertexs and the parallelism info is below.
| vertex | parallelism |
|---|---|
| A | 1 |
| B | 15 |
| C | 20 |
| D | 1 |
| E | 15 |
In this case, the resource sometimes won't balance very good. What I expected is that the vertext B/C/E can distribute evenly amont 5 taskmanagers. Vertex A and D only have 1 parallelism, and it is just some config stream.
Expected allocation strategy: For each vertex, allocate slot evenly among taskmanagers. Then next vertex and repeat. For example, the result below:
| TaskManager1 | TaskManager2 | TaskManager3 | TaskManager4 | TaskManager5 |
|---|---|---|---|---|
| A1 | B1 | B2 | B3 | B4 |
| B5 | B6 | B7 | B8 | B9 |
| B10 | B11 | B12 | B13 | B14 |
| B15 | C1 | C2 | C3 | C4 |
| C5 | C6 | C7 | C8 | C9 |
| C10 | C11 | C12 | C13 | C14 |
| C15 | C16 | C17 | C18 | C19 |
| C20 | D1 | E1 | E2 | E3 |
| E4 | E5 | E6 | E7 | E8 |
| E9 | E10 | E11 | E12 | E13 |
| E14 | E15 | |||
The allocation order is A -> B -> C -> D -> E or some other order, it doesn't matter. The key point is one vertex's all parallel subtasks should be allocated at one time, and then to consider the next vertex. With this strategy, vertex A/D won't influence other vertex's distribution equilibrium.