Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5597 YARN Federation improvements
  3. YARN-8696

[AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.2.0
    • Component/s: nodemanager
    • Labels:
      None

      Description

      Today in FederationInterceptor, the heartbeat to home sub-cluster is synchronous. After the heartbeat is sent out to home sub-cluster, it waits for the home response to come back before merging and returning the (merged) heartbeat result to back AM. If home sub-cluster is suffering from connection issues, or down during an YarnRM master-slave switch, all heartbeat threads in FederationInterceptor will be blocked waiting for home response. As a result, the successful UAM heartbeats from secondary sub-clusters will not be returned to AM at all. Additionally, because of the fact that we kept the same heartbeat responseId between AM and home RM, lots of tricky handling are needed regarding the responseId resync when it comes to FederationInterceptor (part of AMRMProxy, NM) work preserving restart (YARN-6127, YARN-1336), home RM master-slave switch etc.

      In this patch, we change the heartbeat to home sub-cluster to asynchronous, same as the way we handle UAM heartbeats in secondaries. So that any sub-cluster down or connection issues won't impact AM getting responses from other sub-clusters. The responseId is also managed separately for home sub-cluster and AM, and they increment independently. The resync logic becomes much cleaner.

        Attachments

        1. YARN-8696.v1.patch
          116 kB
          Botong Huang
        2. YARN-8696.v2.patch
          117 kB
          Botong Huang
        3. YARN-8696.v3.patch
          86 kB
          Botong Huang
        4. YARN-8696.v4.patch
          85 kB
          Botong Huang
        5. YARN-8696.v5.patch
          86 kB
          Botong Huang
        6. YARN-8696.v6.patch
          85 kB
          Botong Huang
        7. YARN-8696-branch-2.v6.patch
          85 kB
          Botong Huang

        Issue Links

          Activity

            People

            • Assignee:
              botong Botong Huang
              Reporter:
              botong Botong Huang

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment