[YARN-3612] Resource calculation in child tasks is CPU-heavy

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.0
• Labels: BB2015-05-RFC performance
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 23/Jul/12 20:42
• Updated: 07/Jan/17 01:52

Description

In doing some benchmarking on a hadoop-1 derived codebase, I noticed that each of the child tasks was doing a ton of syscalls. Upon stracing, I noticed that it's spending a lot of time looping through all the files in /proc to calculate resource usage.

As a test, I added a flag to disable use of the ResourceCalculatorPlugin within the tasks. On a CPU-bound 500G-sort workload, this improved total job runtime by about 10% (map slot-seconds by 14%, reduce slot seconds by 8%)