[YARN-6715] Fix documentation about NodeHealthScriptRunner

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0, 3.2.2, 3.1.4
• Component/s: documentation, nodemanager
• Hadoop Flags: Reviewed
• Assignee: Peter Bacsko
• Reporter: Peter Bacsko
• Created: 16/Jun/17 18:52
• Updated: 08/Oct/19 15:44
• Resolved: 08/Oct/19 15:44

Description

NodeHealthScriptRunner does not report a bad health if the script exits with an exit code other than 0. Look at the FAILED_WITH_EXIT_CODE case:

    void reportHealthStatus(HealthCheckerExitStatus status) {
      long now = System.currentTimeMillis();
      switch (status) {
      case SUCCESS:
        setHealthStatus(true, "", now);
        break;
      case TIMED_OUT:
        setHealthStatus(false, NODE_HEALTH_SCRIPT_TIMED_OUT_MSG);
        break;
      case FAILED_WITH_EXCEPTION:
        setHealthStatus(false, exceptionStackTrace);
        break;
      case FAILED_WITH_EXIT_CODE:
        setHealthStatus(true, "", now);
        break;
      case FAILED:
        setHealthStatus(false, shexec.getOutput());
        break;
      }
    }

Based on the discussion in YARN-5567, this is intentional, but conflicts with the upstream document, which says:
"If the script exits with a non-zero exit code, times out or results in an exception being thrown, the node is marked as unhealthy"

This statement can be extremely misleading and must be corrected. We might also add an extra comment to reportHealthStatus() which explains that FAILED_WITH_EXIT_CODE is not buggy.

This case also lacks unit test coverage.

Issue Links

• relates to: YARN-5567 Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus