Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11304

Namenode fails to start, even edit log available in the journal node

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.8.0, 2.7.1
    • Fix Version/s: None
    • Component/s: hdfs, journal-node
    • Labels:
    • Environment:

      HDP 2.4.2.0-258

      Description

      JN => JournalNode
      NN => Namenode local directory (dfs.namenode.name.dir)
      Y/N => Is edit file/log present?
      Ex : edits_0000000000001627921-0000000000001627961

      Scenario:

      JN 1 JN 2 JN 3 NN local Is NN started?
      N N Y N Started
      Y N N N Started
      N Y N N Failed
      N Y N Y Started
      Y Y N N Started

      Note: Namenode and JN2 installed on the same machine

      Trace :

      ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
      java.io.IOException: There appears to be a gap in the edit log. We expected txid 1627921, but got txid 1627962.
      at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:215)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)

        Attachments

          Activity

            People

            • Assignee:
              kpalanisamy Karthik Palanisamy
              Reporter:
              kpalanisamy Karthik Palanisamy
            • Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: