[MAPREDUCE-6991] getRogueTaskPID and testProcessTree hangs when the file creation failed

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.0, 2.0.0-alpha, 2.5.0
• Component/s: yarn
• Assignee: Unassigned
• Reporter: John Doe
• Created: 24/Oct/17 21:47
• Updated: 11/May/18 16:41

Description

When writing to file failed, in the following thread, due to disk full, hardware error, etc,

  private class RogueTaskThread extends Thread {
    public void run() {
      try {
        Vector<String> args = new Vector<String>();
        if(isSetsidAvailable()) {
          args.add("setsid");
        }
        args.add("bash");
        args.add("-c");
        args.add(" echo $$ > " + pidFile + "; sh " +
                          shellScript + " " + N + ";") ;
        shexec = new ShellCommandExecutor(args.toArray(new String[0]));
        shexec.execute();
      } catch (ExitCodeException ee) {
        LOG.info("Shell Command exit with a non-zero exit code. This is" +
                 " expected as we are killing the subprocesses of the" +
                 " task intentionally. " + ee);
      } catch (IOException ioe) {
        LOG.info("Error executing shell command " + ioe);
      } finally {
        LOG.info("Exit code: " + shexec.getExitCode());
      }
    }
  }

The getRogueTaskPID() and testProcessTree() get stuck waiting until the file is created or get interrupted.

  private String getRogueTaskPID() {
    File f = new File(pidFile);
    while (!f.exists()) {
      try {
        Thread.sleep(500);
      } catch (InterruptedException ie) {
        break;
      }
    }

    // read from pidFile
    return getPidFromPidFile(pidFile);
  }

  public void testProcessTree() throws Exception {
    // create pid file path string
    tempFile = new File(TEST_ROOT_DIR, getClass().getName() + "_pidFile_" + rm.nextInt() + ".pid");
    tempFile.deleteOnExit();
    pidFile = TEST_ROOT_DIR + File.separator + tempFile.getName();
    .....
    Thread t = new RogueTaskThread();
    t.start();
    String pid = getRogueTaskPID();
    ......
}