Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10251

Both NameNodes could be in STANDBY State if SNN network is unstable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.5.0
    • Component/s: ha
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Following corner scenario happened in one of our cluster.

      1. NN1 was Active and NN2 was Standby
      2. NN2 machine's network was slow
      3. NN1 got shutdown.
      4. NN2 ZKFC got the notification and trying to check for old active for fencing. (This took little more time, again due to slow network)
      5. In between, NN1 got restarted by our automatic monitoring, and ZKFC made it Active.
      6. Now NN2 ZKFC got Old Active as NN1 and it did graceful fencing of NN1 to STANBY.
      7. Before writing ActiveBreadCrumb to ZK, NN2 ZKFC got session timeout and got shutdown before making NN2 Active.

      Now cluster having both NameNodes as STANDBY.
      NN1 ZKFC still thinks that its nameNode is in Active state.
      NN2 ZKFC waiting for election.

        Attachments

        1. HADOOP-10251.patch
          7 kB
          Vinayakumar B
        2. HADOOP-10251.patch
          7 kB
          Vinayakumar B
        3. HADOOP-10251.patch
          9 kB
          Vinayakumar B
        4. HADOOP-10251.patch
          9 kB
          Vinayakumar B
        5. HADOOP-10251.patch
          9 kB
          Vinayakumar B

          Activity

            People

            • Assignee:
              vinayakumarb Vinayakumar B
              Reporter:
              vinayakumarb Vinayakumar B
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: