Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11184

fenced active RM not failing over correctly in HA setup



    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.3
    • None
    • resourcemanager
    • None


      We've observed an issue recently on a production cluster running 3.2.3 in which a fenced Resource Manager remains active, but does not communicate with the ZK state store, and therefore cannot function correctly. This did not occur while running 3.2.2 on the same cluster.

      In more detail, what seems to happen is: 

      1. The active RM gets a NodeExists error from ZK while storing an app in the state store. I suspect that this is caused by some transient connection issue that causes the first node creation request to succeed, but for the response to not reach the RM, triggering a duplicate request which fails with this error.

      2. Because of this error, the active RM is fenced.

      3. Because it is fenced, the active RM starts to transition to standby.

      4. However, the RM never fully transitions to standby. It never logs Transitioning RM to Standby mode from the run method of StandByTransitionRunnable: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java#L1195. Related to this, a jstack of the RM shows that thread being RUNNABLE, but evidently not making progress:

      So the RM doesn't work because it is fenced, but remains active, which causes an outage until a failover is manually initiated.


        1. image-2022-06-14-16-38-00-336.png
          44 kB
          Steven Rand
        2. image-2022-06-14-16-39-50-278.png
          64 kB
          Steven Rand
        3. image-2022-06-14-16-41-39-742.png
          20 kB
          Steven Rand
        4. image-2022-06-14-16-44-45-101.png
          87 kB
          Steven Rand



            Unassigned Unassigned
            Steven Rand Steven Rand
            0 Vote for this issue
            1 Start watching this issue