[TEZ-2612] Critical path analyzer for DAGs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Bikas Saha
• Reporter: Rajesh Balamohan
• Created: 10/Jul/15 13:06
• Updated: 28/Aug/15 01:40

Description

This analyzer plugin/tool can be used to understand the important vertices/tasks of interest in large DAG for perf analysis / finding bottlenecks. It can be used to find out 1.input dependency, 2. failure dependency, 3. scheduling dependency (may be at later stage).

Creating this as a uber ticket. Getting this detail at vertex level might be possible with the existing logs derived from ATS. For task level analysis, certain more details are required.

1. Timeline details like when fetch/merge/compute/sort etc are not captured now. These details can possibly be added in TaskCompletionEvent
2. Need additional details like the last event that completed processing in the input (for tracing at task level)
3. Add downstream task attempt that caused the higher level task to get rescheduled/restarted. This can be used in terms of understanding in cases where the task failed due to read-error.

Issue Links

• is blocked by: TEZ-2752 logUnsuccessful completion in Attempt should write original finish time to ATS

Sub-Tasks

• Allow Input/Processor/Output to specify their execution timeline: Open
• Allow InputInitializers to receive and report based on task attempts: Open
• Add scheduling causal reason to attempts: Open
• Add information about container assignment to attempts: Open