[HDFS-7414] Namenode got shutdown and can't recover where edit update might be missed

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 2.4.1, 2.6.0, 2.5.1
• Assignee: Brahma Reddy Battula
• Reporter: Brahma Reddy Battula
• Created: 20/Nov/14 14:54
• Updated: 09/Feb/15 04:41
• Resolved: 09/Feb/15 04:41

Description

Scenario:
========

Was running mapreduce job.
CPU usage crossed 190% for Datanode and machine became slow..
and seen the following exception ..

Did not get the exact root cause, but as cpu usage more edit log updation might be missed...Need dig to more...anyone have any thoughts.

2014-11-20 05:01:18,430 | ERROR | main | Encountered exception on operation CloseOp [length=0, inodeId=0, path=/outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025, replication=2, mtime=1416409309023, atime=1416409290816, blockSize=67108864, blocks=[blk_1073766144_25321, blk_1073766154_25331, blk_1073766160_25337], permissions=mapred:supergroup:rw-r--r--, aclEntries=null, clientName=, clientMachine=, opCode=OP_CLOSE, txid=162982] | org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:232)
java.io.FileNotFoundException: File does not exist: /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)
2014-11-20 05:01:18,654 | WARN  | main | Encountered exception loading fsimage | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:642)
java.io.FileNotFoundException: File does not exist: /outDir2/_temporary/1/_temporary/attempt_1416390004064_0002_m_000025_1/part-m-00025
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:409)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:224)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:133)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:893)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:640)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:519)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:575)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:741)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1387)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1459)

Issue Links

• is duplicated by: HDFS-7707 Edit log corruption due to delayed block removal again