[HADOOP-14840] Tool to estimate resource requirements of an application pipeline based on prior executions - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.9.0, 3.0.0, 3.1.0
Component/s: tools
Labels:
None

Hadoop Flags:

Reviewed
Release Note:
The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.

Description

We have been working on providing SLAs for job execution on Hadoop. At high level this involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated resources at runtime. The YARN ReservationSystem (~~YARN-1051~~/YARN-2572/~~YARN-5326~~) enable the latter and in this JIRA, we propose to add a tool to Hadoop to predict the resource requirements of a job based on past executions of the job. The system (aka Morpheus) deep dive can be found in our OSDI'16 paper here.