Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5356

NodeManager should communicate physical resource capability to ResourceManager

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently ResourceUtilization contains absolute quantities of resource used (e.g. 4096MB memory used). It would be good if the NM also communicated the actual physical resource capabilities of the node so that the RM can use this data to schedule more effectively (overcommit, etc)

      Currently the only available information is the Resource the node registered with (or later updated using updateNodeResource). However, these aren't really sufficient to get a good view of how utilized a resource is. For example, if a node reports 400% CPU utilization, does that mean it's completely full, or barely utilized? Today there is no reliable way to figure this out.

      Íñigo Goiri - Lots of good work is happening in YARN-2965 so curious if you have thoughts/opinions on this?

        Attachments

        1. YARN-5356.011.patch
          26 kB
          Íñigo Goiri
        2. YARN-5356.010.patch
          23 kB
          Íñigo Goiri
        3. YARN-5356.009.patch
          23 kB
          Íñigo Goiri
        4. YARN-5356.008.patch
          23 kB
          Íñigo Goiri
        5. YARN-5356.007.patch
          20 kB
          Íñigo Goiri
        6. YARN-5356.006.patch
          20 kB
          Íñigo Goiri
        7. YARN-5356.005.patch
          20 kB
          Íñigo Goiri
        8. YARN-5356.004.patch
          20 kB
          Íñigo Goiri
        9. YARN-5356.003.patch
          19 kB
          Íñigo Goiri
        10. YARN-5356.002.patch
          17 kB
          Íñigo Goiri
        11. YARN-5356.002.patch
          17 kB
          Íñigo Goiri
        12. YARN-5356.001.patch
          13 kB
          Íñigo Goiri
        13. YARN-5356.000.patch
          9 kB
          Íñigo Goiri

          Issue Links

            Activity

              People

              • Assignee:
                elgoiri Íñigo Goiri
                Reporter:
                nroberts Nathan Roberts
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: