Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14840

Tool to estimate resource requirements of an application pipeline based on prior executions

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0, 3.1.0
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.

      Description

      We have been working on providing SLAs for job execution on Hadoop. At high level this involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated resources at runtime. The YARN ReservationSystem (YARN-1051/YARN-2572/YARN-5326) enable the latter and in this JIRA, we propose to add a tool to Hadoop to predict the resource requirements of a job based on past executions of the job. The system (aka Morpheus) deep dive can be found in our OSDI'16 paper here.

        Attachments

        1. HADOOP-14840-branch-2-v4.patch
          373 kB
          Subru Krishnan
        2. HADOOP-14840-v1.patch
          344 kB
          Rui Li
        3. HADOOP-14840-v2.patch
          365 kB
          Rui Li
        4. HADOOP-14840-v3.patch
          367 kB
          Rui Li
        5. HADOOP-14840-v4.patch
          373 kB
          Rui Li
        6. ResourceEstimator-design-v1.pdf
          349 kB
          Subru Krishnan

          Issue Links

            Activity

              People

              • Assignee:
                Rui Li Rui Li
                Reporter:
                subru Subru Krishnan
              • Votes:
                0 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: