Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5597 YARN Federation improvements
  3. YARN-8696

[AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.2.0
    • Component/s: nodemanager
    • Labels:
      None

      Description

      Today in FederationInterceptor, the heartbeat to home sub-cluster is synchronous. After the heartbeat is sent out to home sub-cluster, it waits for the home response to come back before merging and returning the (merged) heartbeat result to back AM. If home sub-cluster is suffering from connection issues, or down during an YarnRM master-slave switch, all heartbeat threads in FederationInterceptor will be blocked waiting for home response. As a result, the successful UAM heartbeats from secondary sub-clusters will not be returned to AM at all. Additionally, because of the fact that we kept the same heartbeat responseId between AM and home RM, lots of tricky handling are needed regarding the responseId resync when it comes to FederationInterceptor (part of AMRMProxy, NM) work preserving restart (YARN-6127, YARN-1336), home RM master-slave switch etc.

      In this patch, we change the heartbeat to home sub-cluster to asynchronous, same as the way we handle UAM heartbeats in secondaries. So that any sub-cluster down or connection issues won't impact AM getting responses from other sub-clusters. The responseId is also managed separately for home sub-cluster and AM, and they increment independently. The resync logic becomes much cleaner.

        Attachments

        1. YARN-8696-branch-2.v6.patch
          85 kB
          Botong Huang
        2. YARN-8696.v6.patch
          85 kB
          Botong Huang
        3. YARN-8696.v5.patch
          86 kB
          Botong Huang
        4. YARN-8696.v4.patch
          85 kB
          Botong Huang
        5. YARN-8696.v3.patch
          86 kB
          Botong Huang
        6. YARN-8696.v2.patch
          117 kB
          Botong Huang
        7. YARN-8696.v1.patch
          116 kB
          Botong Huang

          Issue Links

            Activity

              People

              • Assignee:
                botong Botong Huang
                Reporter:
                botong Botong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: