Hadoop Common
  1. Hadoop Common
  2. HADOOP-7458

Namenode not get started! FSNamesystem initialization failed. java.io.FileNotFoundException

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Invalid
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
    • Environment:

      CentOS release 5.5 (Final), 18 node Cluster

    • Tags:
      namenode

      Description

      2011-07-13 12:04:12,967 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
      java.io.FileNotFoundException: File does not exist: /opt/data/tmp/mapred/system/job_201107041958_0120/j^@@@@@^@
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetPermission(FSDirectory.java:544)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:724)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
      2011-07-13 12:04:13,006 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.FileNotFoundException: File does not exist: /opt/data/tmp/mapred/system/job_201107041958_0120/j^@@@@@^@
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetPermission(FSDirectory.java:544)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:724)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
      at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:292)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)

      In the path /opt/data/tmp/mapred, "system/" folder itself is not available

        Activity

        Hide
        Suresh Srinivas added a comment -

        Is this jira still valid?

        Show
        Suresh Srinivas added a comment - Is this jira still valid?
        Hide
        Sakthivel Murugasamy added a comment -

        Hi All,

        Thank you so much for your valuable solutions!

        Problem got resolved, but significant time+data loss(since we were running on an experimental basis, reloaded fewer GB of the data). I used -importCheckpoint option.

        I just would like to tell you the possible scenario/reason of editlog corruption might have happened(correct me if I am wrong),

        Below were the typical configurations in hdfs-site.xml

        • hadoop.tmp.dir : /opt/data/tmp
        • dfs.name.dir : /opt/data/name
        • dfs.data.dir : /opt/data/data
        • mapred.local.dir : $ {hadoop.tmp.dir}/mapred/local

          /opt/data is an mounted storage, size is 50GB. Namenode, SecondaryNamenode( ${hadoop.tmp.dir}

          /dfs/namesecondary) & Datanode directories were configured within /opt/data itself.

        Once I moved 3.6GB compressed(bz2) file, I guess /opt/data memory usage of this dir. could have been 100%(I checked($df -h) after this incident). Then, I ran Hive with simple "Select" query, its job.jar files also needs to be created within the same directory which already has no space. So this is how the editlog corruption could have been occurred.

        This is really a good learning for me! Now I have changed that configurations.

        Thanks

        Show
        Sakthivel Murugasamy added a comment - Hi All, Thank you so much for your valuable solutions! Problem got resolved, but significant time+data loss(since we were running on an experimental basis, reloaded fewer GB of the data). I used -importCheckpoint option. I just would like to tell you the possible scenario/reason of editlog corruption might have happened(correct me if I am wrong), Below were the typical configurations in hdfs-site.xml hadoop.tmp.dir : /opt/data/tmp dfs.name.dir : /opt/data/name dfs.data.dir : /opt/data/data mapred.local.dir : $ {hadoop.tmp.dir}/mapred/local /opt/data is an mounted storage, size is 50GB. Namenode, SecondaryNamenode( ${hadoop.tmp.dir} /dfs/namesecondary) & Datanode directories were configured within /opt/data itself. Once I moved 3.6GB compressed(bz2) file, I guess /opt/data memory usage of this dir. could have been 100%(I checked($df -h) after this incident). Then, I ran Hive with simple "Select" query, its job.jar files also needs to be created within the same directory which already has no space. So this is how the editlog corruption could have been occurred. This is really a good learning for me! Now I have changed that configurations. Thanks
        Hide
        Gerrit Jansen van Vuuren added a comment -

        Hi,

        Yes your right, apologies I did misunderstand you.
        I will file different jira for this.

        Thanks.

        Show
        Gerrit Jansen van Vuuren added a comment - Hi, Yes your right, apologies I did misunderstand you. I will file different jira for this. Thanks.
        Hide
        Uma Maheswara Rao G added a comment -

        Hay Gerrit,
        I think you misunderstood me.
        what i mean is, discussion on this invalid issue may not be effective as Jakob said you could have filed other Jira if we have better recovery mechanism exits ( your approach sounds good). Because people may not look into this Jira that actively because of issue status.

        After validation nothing stops you from putting the namenode out of safe mode.

        you are right!. we can leave safe mode forcibly.

        Show
        Uma Maheswara Rao G added a comment - Hay Gerrit, I think you misunderstood me. what i mean is, discussion on this invalid issue may not be effective as Jakob said you could have filed other Jira if we have better recovery mechanism exits ( your approach sounds good). Because people may not look into this Jira that actively because of issue status. After validation nothing stops you from putting the namenode out of safe mode. you are right!. we can leave safe mode forcibly.
        Hide
        Jakob Homan added a comment -

        @Gerrit
        Sakthivel most likely has a corrupted edits log and his immediate need is get support on how to recover from his particular calamity. The correct forum for that is our mailing lists, where he is already receiving assistance. You're correct that we can always handle error recovery better than we do, and if you have a patch you'd like to offer to help with this, please do so in another JIRA dedicated to your particular fix. Combining discussion of your suggested approach with the troubleshooting needed to address Skithivel's current problem will not be productive.

        Show
        Jakob Homan added a comment - @Gerrit Sakthivel most likely has a corrupted edits log and his immediate need is get support on how to recover from his particular calamity. The correct forum for that is our mailing lists, where he is already receiving assistance. You're correct that we can always handle error recovery better than we do, and if you have a patch you'd like to offer to help with this, please do so in another JIRA dedicated to your particular fix. Combining discussion of your suggested approach with the troubleshooting needed to address Skithivel's current problem will not be productive.
        Hide
        Gerrit Jansen van Vuuren added a comment -

        Of course you can write.
        Once the namenode reads the edit logs and merges the valid entries into the image file your good to go.

        I say put it in safe mode because that way you can validate.
        After validation nothing stops you from putting the namenode out of safe mode.

        The problem is if you don't have a secondary namenode checkpoint because for some reason the secondary namenode failed
        silently. This was my case a couple of days ago, and I managed via the method above to recover all of my data, the cluster is now running writing and reading. With the other alternatives provided in this jira and other mailing lists I would've lost all data.

        Yes I know the issue is invalidated but I strongly disagree with that. Why is the need to recover from a corrupt edits log invalid????? Its not enough to just say: oh you should have had backups.
        Especially when a simple alternative exists, and the namenode has the ability to recover from it.

        I would strongly insist to reopen this issue, especially when a fix does exist.

        Show
        Gerrit Jansen van Vuuren added a comment - Of course you can write. Once the namenode reads the edit logs and merges the valid entries into the image file your good to go. I say put it in safe mode because that way you can validate. After validation nothing stops you from putting the namenode out of safe mode. The problem is if you don't have a secondary namenode checkpoint because for some reason the secondary namenode failed silently. This was my case a couple of days ago, and I managed via the method above to recover all of my data, the cluster is now running writing and reading. With the other alternatives provided in this jira and other mailing lists I would've lost all data. Yes I know the issue is invalidated but I strongly disagree with that. Why is the need to recover from a corrupt edits log invalid????? Its not enough to just say: oh you should have had backups. Especially when a simple alternative exists, and the namenode has the ability to recover from it. I would strongly insist to reopen this issue, especially when a fix does exist.
        Hide
        Uma Maheswara Rao G added a comment -

        But you can not write data right?
        It is better to recover from secondary Namenode in that case.
        if you want only to read the data then you can do this.... (read only cluster?)

        This issue already invalidated here. So, i would suggest to discuss this on mailing list if any thing more on this to discuss.

        Show
        Uma Maheswara Rao G added a comment - But you can not write data right? It is better to recover from secondary Namenode in that case. if you want only to read the data then you can do this.... (read only cluster?) This issue already invalidated here. So, i would suggest to discuss this on mailing list if any thing more on this to discuss.
        Hide
        Gerrit Jansen van Vuuren added a comment -

        This does mean the edit logs is corrupted, but it should be something the namenode can recover from.
        Reasoning is: So you've got 30TB of data. For some reason the metadata is corrupted and the only corruption is last 100 lines in the edit logs.
        Do you need to loose all of your 30TB data because the last 100 lines are corrupt???? This should not be the case.

        A very simple fix is the edit the FSImage.java file and add in a try catch where it loads the edits log.
        The best solution would be that if this happens, the namenode prints out an error, and goes into safemode automatically, but it must still load and not crash.
        I've had this error recently and decided to recompile the hadoop core jar with a simple try catch. It worked and I recovered all of my data, except for some hadoop jobtemp files, which I don't care about any how.

        The patch is:

        Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        ===================================================================
        — src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java (revision 1145902)
        +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java (working copy)
        @@ -1003,12 +1003,20 @@
        int numEdits = 0;
        EditLogFileInputStream edits =
        new EditLogFileInputStream(getImageFile(sd, NameNodeFile.EDITS));
        + try

        { numEdits = FSEditLog.loadFSEdits(edits); + }

        catch(Throwable t)

        { + t.printStackTrace(); + }

        edits.close();
        File editsNew = getImageFile(sd, NameNodeFile.EDITS_NEW);
        if (editsNew.exists() && editsNew.length() > 0) {
        edits = new EditLogFileInputStream(editsNew);
        + try

        { numEdits += FSEditLog.loadFSEdits(edits); + }

        catch(Throwable t)

        { + t.printStackTrace(); + }

        edits.close();
        }

        Show
        Gerrit Jansen van Vuuren added a comment - This does mean the edit logs is corrupted, but it should be something the namenode can recover from. Reasoning is: So you've got 30TB of data. For some reason the metadata is corrupted and the only corruption is last 100 lines in the edit logs. Do you need to loose all of your 30TB data because the last 100 lines are corrupt???? This should not be the case. A very simple fix is the edit the FSImage.java file and add in a try catch where it loads the edits log. The best solution would be that if this happens, the namenode prints out an error, and goes into safemode automatically, but it must still load and not crash. I've had this error recently and decided to recompile the hadoop core jar with a simple try catch. It worked and I recovered all of my data, except for some hadoop jobtemp files, which I don't care about any how. The patch is: Index: src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java =================================================================== — src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java (revision 1145902) +++ src/hdfs/org/apache/hadoop/hdfs/server/namenode/FSImage.java (working copy) @@ -1003,12 +1003,20 @@ int numEdits = 0; EditLogFileInputStream edits = new EditLogFileInputStream(getImageFile(sd, NameNodeFile.EDITS)); + try { numEdits = FSEditLog.loadFSEdits(edits); + } catch(Throwable t) { + t.printStackTrace(); + } edits.close(); File editsNew = getImageFile(sd, NameNodeFile.EDITS_NEW); if (editsNew.exists() && editsNew.length() > 0) { edits = new EditLogFileInputStream(editsNew); + try { numEdits += FSEditLog.loadFSEdits(edits); + } catch(Throwable t) { + t.printStackTrace(); + } edits.close(); }
        Hide
        Sakthivel Murugasamy added a comment -

        Thanks for your reply @Jithendra & @Jacob, I am running hadoop cluster with basic configurations with very minimal custom configurations and haven't taken any backup separately.
        @Jacob, I have written this to hdfs user's list.

        Show
        Sakthivel Murugasamy added a comment - Thanks for your reply @Jithendra & @Jacob, I am running hadoop cluster with basic configurations with very minimal custom configurations and haven't taken any backup separately. @Jacob, I have written this to hdfs user's list.
        Hide
        Jakob Homan added a comment -

        This appears to be an issue of file corruption. The best forum for resolving it is the mailing list, rather than the issue tracker. Please write to the hdfs users' list. Closing.

        Show
        Jakob Homan added a comment - This appears to be an issue of file corruption. The best forum for resolving it is the mailing list, rather than the issue tracker. Please write to the hdfs users' list. Closing.
        Hide
        Jitendra Nath Pandey added a comment -

        The file name "/opt/data/tmp/mapred/system/job_201107041958_0120/...." seems to be not right. Probably the edit logs got corrupted. Do you have a backup for edit logs?

        Show
        Jitendra Nath Pandey added a comment - The file name "/opt/data/tmp/mapred/system/job_201107041958_0120/...." seems to be not right. Probably the edit logs got corrupted. Do you have a backup for edit logs?
        Hide
        Sakthivel Murugasamy added a comment -

        In the path /opt/data/tmp/mapred, "system/" folder itself is not available

        Show
        Sakthivel Murugasamy added a comment - In the path /opt/data/tmp/mapred, "system/" folder itself is not available

          People

          • Assignee:
            Unassigned
            Reporter:
            Sakthivel Murugasamy
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 48h
              48h
              Remaining:
              Remaining Estimate - 48h
              48h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development