Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3923

Introduce adaptive heartbeat between RM and AM

    XMLWordPrintableJSON

    Details

      Description

      Currently, the communication between RM and AM are based on pull-based heartbeat protocol. Along with AM heartbeat, AM sends the container requests to RM and retrieves the new allocated containers. Also, it will update RM's view of resource requests of applications. How frequently AM sends the heartbeat will impact the task throughput and latency of YARN scheduler. Although the heartbeat interval can be configured in yarn-stie.xml (mapred-site.xml for MR AppMaster), if it is configured too short, it will increase the load of RM and bring unnecessary overhead.

      We propose the adaptive heartbeat between RM and AM to achieve a balance between updating AM's info promptly and minimizing the overhead of extra heartbeats. With adaptive heartbeat, AM still honors the current heartbeat interval and sends the heartbeat regularly. However, a heartbeat is triggered as soon as a container request is made. To prevent AM sending heartbeat too frequently, a minimum interval can be used to throttle the frequency of heartbeat.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                xiaodike Xiaodi Ke
              • Votes:
                0 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated: