[TEZ-3426] Second AM attempt launched for session mode and recovery disabled for certain cases

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.7.2, 0.9.0, 0.8.5
• Target Version/s: 0.7.2
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: Jonathan Turner Eagles
• Created: 01/Sep/16 20:13
• Updated: 14/Mar/17 03:49
• Resolved: 06/Sep/16 22:47

Description

ApplicationSubmissionContext#setMaxAppAttempts does not fully guarantee that there will be only that many attempts at maximum. There are a few exceptional cases that are not count. Tez should protect itself from accidentally starting the second attempt in session mode and when recovery is disabled since the second attempt will always succeed with no work to do.

  @Override
  public boolean shouldCountTowardsMaxAttemptRetry() {
    try {
      this.readLock.lock();
      int exitStatus = getAMContainerExitStatus();
      return !(exitStatus == ContainerExitStatus.PREEMPTED
          || exitStatus == ContainerExitStatus.ABORTED
          || exitStatus == ContainerExitStatus.DISKS_FAILED
          || exitStatus == ContainerExitStatus.KILLED_BY_RESOURCEMANAGER);
    } finally {
      this.readLock.unlock();
    }
  }

Issue Links

• is related to: YARN-5617 AMs only intended to run one attempt can be run more than once
• relates to: YARN-614 Separate AM failures from hardware failure or YARN error and do not count them to AM retry count