Details
-
Bug
-
Status: Patch Available
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Tez task attempt start times don't reflect time spent in localization.
In the MapReduce framework, the time spent in localization was included in the total runtime of each task attempt. But since Tez reuses containers, the time spent localizing for a container is not captured. The start time of the first attempt in that container will only be set after the localization has completed.
The result is that attempts can appear as if they are not being run even though there are resources available in the queue. An attempt can be assigned to a container, but if the container is on a slow node and it takes a long time to localize, the attempt state will remain pending until localization completes.
The impact risk is that tasks will not speculate during localization since they haven't started