[YARN-3480] Recovery may get very slow with lots of services with lots of app-attempts

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: resourcemanager
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Assignee: Jun Gong
• Reporter: Jun Gong
• Created: 13/Apr/15 16:57
• Updated: 30/Aug/16 04:24
• Resolved: 30/Dec/15 03:00

Description

When RM HA is enabled and running containers are kept across attempts, apps are more likely to finish successfully with more retries(attempts), so it will be better to set 'yarn.resourcemanager.am.max-attempts' larger. However it will make RMStateStore(FileSystem/HDFS/ZK) store more attempts, and make RM recover process much slower. It might be better to set max attempts to be stored in RMStateStore.

BTW: When 'attemptFailuresValidityInterval'(introduced in YARN-611) is set to a small value, retried attempts might be very large. So we need to delete some attempts stored in RMStateStore and RMStateStore.

Issue Links

• is blocked by: YARN-1039 Add parameter for YARN resource requests to indicate "long lived"
• is related to: YARN-4497 RM might fail to restart when recovering apps whose attempts are missing
• is related to: YARN-4584 RM startup failure when AM attempts greater than max-attempts
• is related to: YARN-3668 Long run service shouldn't be killed even if Yarn crashed
• relates to: YARN-4929 Explore a better way than sleeping for a while in some test cases