Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
Description
The deploy of tasks will take long time when we submit a high parallelism job. And Execution#deploy run in mainThread, so it will block JobMaster process other akka messages, such as Heartbeat. The creation of TaskDeploymentDescriptor take most of time. We can put the creation in future.
For example, A job [source(8000)->sink(8000)], the total 16000 tasks from SCHEDULED to DEPLOYING took more than 1mins. This caused the heartbeat of TaskManager timeout and job never success.
Attachments
Attachments
Issue Links
- duplicates
-
FLINK-23005 Cache the compressed serialized value of ShuffleDescriptors
-
- Closed
-
- relates to
-
FLINK-23005 Cache the compressed serialized value of ShuffleDescriptors
-
- Closed
-