[YARN-9413] Queue resource leak after app fail for CapacityScheduler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Fix Version/s: 3.0.4, 3.3.0, 3.2.1, 3.1.3
• Component/s: capacityscheduler
• Hadoop Flags: Reviewed
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 27/Mar/19 04:06
• Updated: 08/Apr/19 05:46
• Resolved: 08/Apr/19 05:45

Description

To reproduce this problem:

1. Submit an app which is configured to keep containers across app attempts and should fail after AM finished at first time (am-max-attempts=1).
2. App is started with 2 containers running on NM1 node.
3. Fail the AM of the application with PREEMPTED exit status which should not count towards max attempt retry but app will fail immediately.
4. Used resource of this queue leaks after app fail.

The root cause is the inconsistency of handling app attempt failure between RMAppAttemptImpl$BaseFinalTransition#transition and RMAppImpl$AttemptFailedTransition#transition:

1. After app fail, RMAppFailedAttemptEvent will be sent in RMAppAttemptImpl$BaseFinalTransition#transition, if exit status of AM container is PREEMPTED/ABORTED/DISKS_FAILED/KILLED_BY_RESOURCEMANAGER, it will not count towards max attempt retry, so that it will send AppAttemptRemovedSchedulerEvent with keepContainersAcrossAppAttempts=true and RMAppFailedAttemptEvent with transferStateFromPreviousAttempt=true.
2. RMAppImpl$AttemptFailedTransition#transition handle RMAppFailedAttemptEvent and will fail the app if its max app attempts is 1.
3. CapacityScheduler handles AppAttemptRemovedSchedulerEvent in CapcityScheduler#doneApplicationAttempt, it will skip killing and calling completion process for containers belong to this app, so that queue resource leak happens.