[TEZ-1512] VertexImpl.getTask(int) can be CPU intensive when lots of tasks are present in the vertex

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Labels: performance
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 28/Aug/14 05:32
• Updated: 06/Sep/14 01:35
• Resolved: 28/Aug/14 08:15

Description

I tried a synthetic benchmark (without much input data) with the tez app. This was tried to understand the bare minimum time taken by Tez for container launch / reuse / scheduling etc.

Profiling DAGAppMaster showed that lots of CPU time was spent on VertexImpl.getTask(int) which gets accessed as a part of event handling and transitions.

This problem would more prevalent in large jobs which has got lots of small tasks.

I will attach the perf SVG output of the DAG soon.