Hadoop YARN
  1. Hadoop YARN
  2. YARN-415

Capture aggregate memory allocation at the app-level for chargeback

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.6.0
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      For the purpose of chargeback, I'd like to be able to compute the cost of an
      application in terms of cluster resource usage. To start out, I'd like to get the memory utilization of an application. The unit should be MB-seconds or something similar and, from a chargeback perspective, the memory amount should be the memory reserved for the application, as even if the app didn't use all that memory, no one else was able to use it.

      (reserved ram for container 1 * lifetime of container 1) + (reserved ram for
      container 2 * lifetime of container 2) + ... + (reserved ram for container n * lifetime of container n)

      It'd be nice to have this at the app level instead of the job level because:
      1. We'd still be able to get memory usage for jobs that crashed (and wouldn't appear on the job history server).
      2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).

      This new metric should be available both through the RM UI and RM Web Services REST API.

      1. YARN-415.201409102216.txt
        121 kB
        Eric Payne
      2. YARN-415.201409092204.txt
        120 kB
        Eric Payne
      3. YARN-415.201409040036.txt
        120 kB
        Eric Payne
      4. YARN-415.201408212033.txt
        118 kB
        Eric Payne
      5. YARN-415.201408181938.txt
        119 kB
        Eric Payne
      6. YARN-415.201408181938.txt
        119 kB
        Eric Payne
      7. YARN-415.201408150030.txt
        119 kB
        Eric Payne
      8. YARN-415.201408132109.txt
        120 kB
        Eric Payne
      9. YARN-415.201408092006.txt
        116 kB
        Eric Payne
      10. YARN-415.201408080204.txt
        121 kB
        Eric Payne
      11. YARN-415.201408062232.txt
        113 kB
        Eric Payne
      12. YARN-415.201407281816.txt
        99 kB
        Eric Payne
      13. YARN-415.201407242148.txt
        79 kB
        Eric Payne
      14. YARN-415.201407232237.txt
        78 kB
        Eric Payne
      15. YARN-415.201407172144.txt
        71 kB
        Eric Payne
      16. YARN-415.201407171553.txt
        85 kB
        Eric Payne
      17. YARN-415.201407071542.txt
        83 kB
        Eric Payne
      18. YARN-415.201407042037.txt
        83 kB
        Eric Payne
      19. YARN-415.201406262136.txt
        81 kB
        Eric Payne
      20. YARN-415.201406031616.txt
        83 kB
        Eric Payne
      21. YARN-415.201405311749.txt
        83 kB
        Eric Payne
      22. YARN-415--n10.patch
        103 kB
        Andrey Klochkov
      23. YARN-415--n9.patch
        59 kB
        Andrey Klochkov
      24. YARN-415--n8.patch
        63 kB
        Andrey Klochkov
      25. YARN-415--n7.patch
        59 kB
        Andrey Klochkov
      26. YARN-415--n6.patch
        59 kB
        Andrey Klochkov
      27. YARN-415--n5.patch
        55 kB
        Andrey Klochkov
      28. YARN-415--n4.patch
        55 kB
        Andrey Klochkov
      29. YARN-415--n3.patch
        60 kB
        Andrey Klochkov
      30. YARN-415--n2.patch
        57 kB
        Andrey Klochkov
      31. YARN-415.patch
        57 kB
        Andrey Klochkov

        Issue Links

          Activity

          Kendall Thrapp created issue -
          Andrey Klochkov made changes -
          Field Original Value New Value
          Assignee Andrey Klochkov [ aklochkov ]
          Andrey Klochkov made changes -
          Attachment YARN-415.patch [ 12604125 ]
          Andrey Klochkov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n2.patch [ 12604132 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n3.patch [ 12604142 ]
          Omkar Vinit Joshi made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n4.patch [ 12606923 ]
          Jason Lowe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.3.0 [ 12324589 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n5.patch [ 12607275 ]
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n6.patch [ 12607895 ]
          Andrey Klochkov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n7.patch [ 12609031 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n8.patch [ 12609068 ]
          Andrey Klochkov made changes -
          Link This issue depends upon YARN-1335 [ YARN-1335 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n9.patch [ 12610399 ]
          Andrey Klochkov made changes -
          Attachment YARN-415--n10.patch [ 12610446 ]
          Eric Payne made changes -
          Attachment YARN-415.201405311749.txt [ 12648154 ]
          Eric Payne made changes -
          Attachment YARN-415.201406031616.txt [ 12648183 ]
          Eric Payne made changes -
          Attachment YARN-415.201406262136.txt [ 12652731 ]
          Eric Payne made changes -
          Attachment YARN-415.201407042037.txt [ 12654144 ]
          Eric Payne made changes -
          Attachment YARN-415.201407071542.txt [ 12654334 ]
          Eric Payne made changes -
          Attachment YARN-415.201407171553.txt [ 12656291 ]
          Eric Payne made changes -
          Attachment YARN-415.201407172144.txt [ 12656361 ]
          Eric Payne made changes -
          Attachment YARN-415.201407232237.txt [ 12657484 ]
          Eric Payne made changes -
          Attachment YARN-415.201407242148.txt [ 12657692 ]
          Eric Payne made changes -
          Attachment YARN-415.201407281816.txt [ 12658211 ]
          Eric Payne made changes -
          Attachment YARN-415.201408062232.txt [ 12660287 ]
          Eric Payne made changes -
          Attachment YARN-415.201408080204.txt [ 12660535 ]
          Eric Payne made changes -
          Attachment YARN-415.201408092006.txt [ 12660822 ]
          Eric Payne made changes -
          Attachment YARN-415.201408132109.txt [ 12661548 ]
          Eric Payne made changes -
          Attachment YARN-415.201408150030.txt [ 12661961 ]
          Eric Payne made changes -
          Attachment YARN-415.201408181938.txt [ 12662572 ]
          Eric Payne made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Affects Version/s 0.23.6 [ 12323501 ]
          Target Version/s
          Eric Payne made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.5.0 [ 12326262 ]
          Target Version/s 3.0.0, 2.6.0 [ 12323268, 12327197 ]
          Eric Payne made changes -
          Attachment YARN-415.201408181938.txt [ 12663246 ]
          Kendall Thrapp made changes -
          Summary Capture memory utilization at the app-level for chargeback Capture memory allocation at the app-level for chargeback
          Karthik Kambatla (Inactive) made changes -
          Summary Capture memory allocation at the app-level for chargeback Capture aggregate memory allocation at the app-level for chargeback
          Eric Payne made changes -
          Attachment YARN-415.201408212033.txt [ 12663542 ]
          Eric Payne made changes -
          Attachment YARN-415.201409040036.txt [ 12666493 ]
          Eric Payne made changes -
          Attachment YARN-415.201409092204.txt [ 12667510 ]
          Eric Payne made changes -
          Attachment YARN-415.201409102216.txt [ 12667877 ]
          Jian He made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.6.0 [ 12327197 ]
          Resolution Fixed [ 1 ]
          Andrey Klochkov made changes -
          Assignee Andrey Klochkov [ aklochkov ] Eric Payne [ eepayne ]
          Zhijie Shen made changes -
          Link This issue is broken by YARN-2542 [ YARN-2542 ]
          Zhijie Shen made changes -
          Link This issue is broken by YARN-2542 [ YARN-2542 ]
          Zhijie Shen made changes -
          Link This issue breaks YARN-2542 [ YARN-2542 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Xuan Gong made changes -
          Link This issue duplicates YARN-1899 [ YARN-1899 ]

            People

            • Assignee:
              Eric Payne
              Reporter:
              Kendall Thrapp
            • Votes:
              0 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development