Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-415

Capture aggregate memory allocation at the app-level for chargeback

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.5.0
    • 2.6.0
    • resourcemanager
    • None
    • Reviewed

    Description

      For the purpose of chargeback, I'd like to be able to compute the cost of an
      application in terms of cluster resource usage. To start out, I'd like to get the memory utilization of an application. The unit should be MB-seconds or something similar and, from a chargeback perspective, the memory amount should be the memory reserved for the application, as even if the app didn't use all that memory, no one else was able to use it.

      (reserved ram for container 1 * lifetime of container 1) + (reserved ram for
      container 2 * lifetime of container 2) + ... + (reserved ram for container n * lifetime of container n)

      It'd be nice to have this at the app level instead of the job level because:
      1. We'd still be able to get memory usage for jobs that crashed (and wouldn't appear on the job history server).
      2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).

      This new metric should be available both through the RM UI and RM Web Services REST API.

      Attachments

        1. YARN-415.201405311749.txt
          83 kB
          Eric Payne
        2. YARN-415.201406031616.txt
          83 kB
          Eric Payne
        3. YARN-415.201406262136.txt
          81 kB
          Eric Payne
        4. YARN-415.201407042037.txt
          83 kB
          Eric Payne
        5. YARN-415.201407071542.txt
          83 kB
          Eric Payne
        6. YARN-415.201407171553.txt
          85 kB
          Eric Payne
        7. YARN-415.201407172144.txt
          71 kB
          Eric Payne
        8. YARN-415.201407232237.txt
          78 kB
          Eric Payne
        9. YARN-415.201407242148.txt
          79 kB
          Eric Payne
        10. YARN-415.201407281816.txt
          99 kB
          Eric Payne
        11. YARN-415.201408062232.txt
          113 kB
          Eric Payne
        12. YARN-415.201408080204.txt
          121 kB
          Eric Payne
        13. YARN-415.201408092006.txt
          116 kB
          Eric Payne
        14. YARN-415.201408132109.txt
          120 kB
          Eric Payne
        15. YARN-415.201408150030.txt
          119 kB
          Eric Payne
        16. YARN-415.201408181938.txt
          119 kB
          Eric Payne
        17. YARN-415.201408181938.txt
          119 kB
          Eric Payne
        18. YARN-415.201408212033.txt
          118 kB
          Eric Payne
        19. YARN-415.201409040036.txt
          120 kB
          Eric Payne
        20. YARN-415.201409092204.txt
          120 kB
          Eric Payne
        21. YARN-415.201409102216.txt
          121 kB
          Eric Payne
        22. YARN-415.patch
          57 kB
          Andrey Klochkov
        23. YARN-415--n10.patch
          103 kB
          Andrey Klochkov
        24. YARN-415--n2.patch
          57 kB
          Andrey Klochkov
        25. YARN-415--n3.patch
          60 kB
          Andrey Klochkov
        26. YARN-415--n4.patch
          55 kB
          Andrey Klochkov
        27. YARN-415--n5.patch
          55 kB
          Andrey Klochkov
        28. YARN-415--n6.patch
          59 kB
          Andrey Klochkov
        29. YARN-415--n7.patch
          59 kB
          Andrey Klochkov
        30. YARN-415--n8.patch
          63 kB
          Andrey Klochkov
        31. YARN-415--n9.patch
          59 kB
          Andrey Klochkov

        Issue Links

          Activity

            People

              epayne Eric Payne
              kthrapp Kendall Thrapp
              Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: