[MAPREDUCE-6867] ApplicationMaster hung on OOM Error

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: applicationmaster
• Assignee: Bilwa S T
• Reporter: Bilwa S T
• Created: 25/Mar/17 08:25
• Updated: 20/Mar/18 04:14

Description

Whenever OOM Error is thown, YarnUncaughtExceptionHandler will call ExitUtil.halt(-1).But while halting, OOM might occur which is not handled.

We came across a scenario where in when we submit mapreduce application ,OOM error occured in committerEventProcessor and then AM did not halt and did not log the following.Finally AM got hang since it's not thrown to main thread.

LOG.info("Halt with status " + status + " Message: " + msg);

org.apache.hadoop.util.ExitUtil.halt(int, String)

public static void halt(int status, String msg) throws HaltException {
    LOG.info("Halt with status " + status + " Message: " + msg);
    if (systemHaltDisabled) {
      HaltException ee = new HaltException(status, msg);
      LOG.fatal("Halt called", ee);
      if (null == firstHaltException) {
        firstHaltException = ee;
      }
      throw ee;
    }
    Runtime.getRuntime().halt(status);
  }