Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14840

Tool to estimate resource requirements of an application pipeline based on prior executions

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0, 3.1.0
    • tools
    • None
    • Reviewed
    • The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.

    Description

      We have been working on providing SLAs for job execution on Hadoop. At high level this involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated resources at runtime. The YARN ReservationSystem (YARN-1051/YARN-2572/YARN-5326) enable the latter and in this JIRA, we propose to add a tool to Hadoop to predict the resource requirements of a job based on past executions of the job. The system (aka Morpheus) deep dive can be found in our OSDI'16 paper here.

      Attachments

        1. ResourceEstimator-design-v1.pdf
          349 kB
          Subramaniam Krishnan
        2. HADOOP-14840-v4.patch
          373 kB
          Rui Li
        3. HADOOP-14840-v3.patch
          367 kB
          Rui Li
        4. HADOOP-14840-v2.patch
          365 kB
          Rui Li
        5. HADOOP-14840-v1.patch
          344 kB
          Rui Li
        6. HADOOP-14840-branch-2-v4.patch
          373 kB
          Subramaniam Krishnan

        Issue Links

          Activity

            People

              Rui Li Rui Li
              subru Subramaniam Krishnan
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: