Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-415

Capture aggregate memory allocation at the app-level for chargeback

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.6.0
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For the purpose of chargeback, I'd like to be able to compute the cost of an
      application in terms of cluster resource usage. To start out, I'd like to get the memory utilization of an application. The unit should be MB-seconds or something similar and, from a chargeback perspective, the memory amount should be the memory reserved for the application, as even if the app didn't use all that memory, no one else was able to use it.

      (reserved ram for container 1 * lifetime of container 1) + (reserved ram for
      container 2 * lifetime of container 2) + ... + (reserved ram for container n * lifetime of container n)

      It'd be nice to have this at the app level instead of the job level because:
      1. We'd still be able to get memory usage for jobs that crashed (and wouldn't appear on the job history server).
      2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).

      This new metric should be available both through the RM UI and RM Web Services REST API.

        Attachments

        1. YARN-415--n9.patch
          59 kB
          Andrey Klochkov
        2. YARN-415--n8.patch
          63 kB
          Andrey Klochkov
        3. YARN-415--n7.patch
          59 kB
          Andrey Klochkov
        4. YARN-415--n6.patch
          59 kB
          Andrey Klochkov
        5. YARN-415--n5.patch
          55 kB
          Andrey Klochkov
        6. YARN-415--n4.patch
          55 kB
          Andrey Klochkov
        7. YARN-415--n3.patch
          60 kB
          Andrey Klochkov
        8. YARN-415--n2.patch
          57 kB
          Andrey Klochkov
        9. YARN-415--n10.patch
          103 kB
          Andrey Klochkov
        10. YARN-415.patch
          57 kB
          Andrey Klochkov
        11. YARN-415.201409102216.txt
          121 kB
          Eric Payne
        12. YARN-415.201409092204.txt
          120 kB
          Eric Payne
        13. YARN-415.201409040036.txt
          120 kB
          Eric Payne
        14. YARN-415.201408212033.txt
          118 kB
          Eric Payne
        15. YARN-415.201408181938.txt
          119 kB
          Eric Payne
        16. YARN-415.201408181938.txt
          119 kB
          Eric Payne
        17. YARN-415.201408150030.txt
          119 kB
          Eric Payne
        18. YARN-415.201408132109.txt
          120 kB
          Eric Payne
        19. YARN-415.201408092006.txt
          116 kB
          Eric Payne
        20. YARN-415.201408080204.txt
          121 kB
          Eric Payne
        21. YARN-415.201408062232.txt
          113 kB
          Eric Payne
        22. YARN-415.201407281816.txt
          99 kB
          Eric Payne
        23. YARN-415.201407242148.txt
          79 kB
          Eric Payne
        24. YARN-415.201407232237.txt
          78 kB
          Eric Payne
        25. YARN-415.201407172144.txt
          71 kB
          Eric Payne
        26. YARN-415.201407171553.txt
          85 kB
          Eric Payne
        27. YARN-415.201407071542.txt
          83 kB
          Eric Payne
        28. YARN-415.201407042037.txt
          83 kB
          Eric Payne
        29. YARN-415.201406262136.txt
          81 kB
          Eric Payne
        30. YARN-415.201406031616.txt
          83 kB
          Eric Payne
        31. YARN-415.201405311749.txt
          83 kB
          Eric Payne

          Issue Links

            Activity

              People

              • Assignee:
                eepayne Eric Payne
                Reporter:
                kthrapp Kendall Thrapp
              • Votes:
                0 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: