Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10475

Scale RM-NM heartbeat interval based on node utilization

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.10.1, 3.4.0
    • Fix Version/s: 3.2.2, 3.4.0, 3.3.1, 3.1.5, 3.2.3
    • Component/s: yarn
    • Labels:
      None

      Description

      Add the ability to scale the RM-NM heartbeat interval based on node cpu utilization compared to overall cluster cpu utilization.  If a node is over-utilized compared to the rest of the cluster, it's heartbeat interval slows down.  If it is under-utilized compared to the rest of the cluster, it's heartbeat interval speeds up.

      This is a feature we have been running with internally in production for several years.  It was developed by Nathan Roberts, based on the observation that larger faster nodes on our cluster were under-utilized compared to smaller slower nodes. 

      This feature is dependent on YARN-10450, which added cluster-wide utilization metrics.

        Attachments

        1. YARN-10475-branch-3.3.003.patch
          35 kB
          Jim Brennan
        2. YARN-10475-branch-3.2.003.patch
          35 kB
          Jim Brennan
        3. YARN-10475.003.patch
          35 kB
          Jim Brennan
        4. YARN-10475.002.patch
          32 kB
          Jim Brennan
        5. YARN-10475.001.patch
          31 kB
          Jim Brennan

          Activity

            People

            • Assignee:
              Jim_Brennan Jim Brennan
              Reporter:
              Jim_Brennan Jim Brennan
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: