[YARN-3813] Support Application timeout feature in YARN.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0
• Component/s: scheduler
• Target Version/s: 3.1.0
• Hadoop Flags: Reviewed
• Assignee: Rohith Sharma K S
• Reporter: nijel
• Created: 17/Jun/15 09:47
• Updated: 08/Aug/18 04:05
• Resolved: 30/Oct/17 07:09

Description

It will be useful to support Application Timeout in YARN. Some use cases are not worried about the output of the applications if the application is not completed in a specific time.

Background:
The requirement is to show the CDR statistics of last few minutes, say for every 5 minutes. The same Job will run continuously with different dataset.
So one job will be started in every 5 minutes. The estimate time for this task is 2 minutes or lesser time.
If the application is not completing in the given time the output is not useful.

Proposal
So idea is to support application timeout, with which timeout parameter is given while submitting the job.
Here, user is expecting to finish (complete or kill) the application in the given time.

One option for us is to move this logic to Application client (who submit the job).
But it will be nice if it can be generic logic and can make more robust.

Kindly provide your suggestions/opinion on this feature. If it sounds good, i will update the design doc and prototype patch

Issue Links

• duplicates: YARN-8573 Add Maximum lifetime for YARN applications
• is related to: YARN-2266 Add an application timeout service in RM to kill applications which are not getting resources
• is related to: YARN-5692 Support more application timeout types
• relates to: AMBARI-24387 Support YARN Application timeout feature in Ambari Capacity Scheduler View
• relates to: YARN-2487 Need to support timeout of AM When no containers are assigned to it for a defined period
• relates to: MAPREDUCE-6490 Facilitate the application life time out in mapreduce applications.

Sub-Tasks

• Add a service for monitoring application life time out: Resolved
• Add Application timeouts in Application report and CLI: Resolved
• Provide an API to update lifetime of an application.: Resolved
• Add REST endpoints for updating application timeouts: Resolved
• Render Application Timeout value in web UI: Resolved
• Capability to register for a notification/callback on the expiry of timeouts: Resolved
• Document timeout interfaces CLI and REST APIs: Resolved
• Retrospect ApplicationReport#getApplicationTimeouts: Resolved
• Validate application update timeout request parameters: Resolved
• RM fails to start during an upgrade - Failed to load/recover state (YarnException: Invalid application timeout, value=0 for type=LIFETIME): Resolved
• Introduce default and max lifetime of application at LeafQueue level: Resolved
• Application lifetime does not work with FairScheduler: Resolved