Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5597 YARN Federation improvements
  3. YARN-7720

Race condition between second app attempt and UAM timeout when first attempt node is down

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In Federation, multiple attempts of an application share the same UAM in each secondary sub-cluster. When first attempt fails, we reply on the fact that secondary RM won't kill the existing UAM before the AM heartbeat timeout (default at 10 min). When second attempt comes up in the home sub-cluster, it will pick up the UAM token from Yarn Registry and resume the UAM heartbeat to secondary RMs.

      The default heartbeat timeout for NM and AM are both 10 mins. The problem is that when the first attempt node goes down or out of connection, only after 10 mins will the home RM mark the first attempt as failed, and then schedule the 2nd attempt in some other node. By then the UAMs in secondaries are already timing out, and they might not survive until the second attempt comes up.

        Attachments

        1. YARN-7720.v2.patch
          4 kB
          Botong Huang
        2. YARN-7720.v1.patch
          4 kB
          Botong Huang

          Activity

            People

            • Assignee:
              botong Botong Huang
              Reporter:
              botong Botong Huang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: