Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9459

ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.2-alpha
    • 2.1.0-beta
    • ha
    • None
    • Reviewed

    Description

      ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE.

      1. ZKFC restarted.
      2. due to machine busy, first zk connection is expired even before the health monitoring returned the status.
      3. On re-establishment transitionToActive will be called, at this time appData will be null,
      4. So now ActiveBreadCrumb will have null.
      5. After this any failovers will fail throwing

      java.lang.NullPointerException
      	at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171)
      	at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892)
      	at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797)
      	at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475)
      	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545)
      	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)

      Should not join the election before service is HEALTHY

      Attachments

        1. hdfs-4463.txt
          3 kB
          Todd Lipcon
        2. HDFS-4463.patch
          2 kB
          Vinayakumar B

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vinayakumarb Vinayakumar B
            vinayakumarb Vinayakumar B
            Votes:
            0 Vote for this issue
            Watchers:
            13 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment