Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
When a task gets scheduled then we dont know what caused it. Some possibilities are
1) initial scheduling by the vertex manager - causality determined by VM. E.g. dynamic partition pruning VM in Hive can point causality to the attempt that sent it the stats needed to complete the partition pruning logic.
2) re-scheduling due to own previous version failure - causality points to the previous version that just failed
3) re-scheduling because read error reported by consumer - causality points to the consumer attempt that reported the error and caused the scheduling.
This causality relationship can be used to stitch together scheduling dependencies in the execution timeline of the DAG.