Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14336 Procedure V2 Phase 1 - Procedure Framework and Making DDL Operations fault tolerant
  3. HBASE-16056

Procedure v2 - fix master crash for FileNotFound

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.3.0, 1.2.1, 1.1.5, 2.0.0
    • 1.3.0, 1.2.2, 1.1.6, 2.0.0
    • proc-v2
    • None

    Description

      Stephen Yuan Jiang and Zhihong Yu reported a backup master not able to start with FileNotFound during proc-v2 lease recovery. (another restart should have solved the problem)

      FileNotFoundException: File does not exist: /hbase/MasterProcWALs/state-000001.log
namenode.INodeFile.valueOf(INodeFile.java:61) at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671)

      this may happen when the other master is still active (e.g. GC) and tries to remove files while the other master tries to become active. This operation is retryable so the code should able to handle that.

      Attachments

        1. HBASE-16056-v2.patch
          5 kB
          Matteo Bertozzi
        2. HBASE-16056-v1.patch
          4 kB
          Matteo Bertozzi
        3. HBASE-16056-v0.patch
          3 kB
          Matteo Bertozzi

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mbertozzi Matteo Bertozzi
            mbertozzi Matteo Bertozzi
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment