Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7414

Namenode got shutdown and can't recover where edit update might be missed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 2.4.1, 2.6.0, 2.5.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Scenario:
      ========

      Was running mapreduce job.
      CPU usage crossed 190% for Datanode and machine became slow..
      and seen the following exception ..

      Did not get the exact root cause, but as cpu usage more edit log updation might be missed...Need dig to more...anyone have any thoughts.

      2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation CloseOp [length=0, inodeId=0, path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025, replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864, blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337], permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=, clientMachine=, opCode=OP_CLOSE, txid=162982] | org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
      java.io.FileNotFoundException: File does not exist: /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
              at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
              at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
      2014-11-20 05:01:18,654 | WARN  | main | Encountered exception loading fsimage | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
      java.io.FileNotFoundException: File does not exist: /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
              at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
              at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
              at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
              at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
      
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brahmareddy Brahma Reddy Battula
                Reporter:
                brahmareddy Brahma Reddy Battula
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: