Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.5.2
-
None
-
None
-
Reviewed
Description
ResourceCalculatorProcessTree scraps all of /proc/ for PIDs which are part of the current task's process group.
This is mostly wasted in Tez, since unlike YARN which has to do this since it has the PID for the container-executor process (bash) and has to trace the bash -> java spawn inheritance.
The latency effect of this is less clearly visible with the profiler turned on as this is primarily related to rate of syscalls + overhead in the kernel (via the following codepath in YARN).
private List<String> getProcessList() { String[] processDirs = (new File(procfsDir)).list(); ... for (String dir : processDirs) { try { if ((new File(procfsDir, dir)).isDirectory()) { processList.add(dir); } ... public void updateProcessTree() { if (!pid.equals(deadPid)) { // Get the list of processes List<String> processList = getProcessList(); ... for (String proc : processList) { // Get information for each process ProcessInfo pInfo = new ProcessInfo(proc); if (constructProcessInfo(pInfo, procfsDir) != null) { allProcessInfo.put(proc, pInfo); if (proc.equals(this.pid)) { me = pInfo; // cache 'me' processTree.put(proc, pInfo); } } }