Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1594

When the disk becomes full Namenode is getting shutdown and not able to recover

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.21.1, 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Environment:

      Linux linux124 2.6.27.19-5-default #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.
      Show
      Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.

      Description

      When the disk becomes full name node is shutting down and if we try to start after making the space available It is not starting and throwing the below exception.

       
      
      2011-01-24 23:23:33,727 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
      java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:180)
      	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:117)
      	at org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.readString(FSImageSerialization.java:201)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:60)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1089)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1041)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:487)
      	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:149)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:306)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:284)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:328)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:356)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:577)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:570)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1538)
      2011-01-24 23:23:33,729 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:180)
      	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:117)
      	at org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.readString(FSImageSerialization.java:201)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:60)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1089)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1041)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:487)
      	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:149)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:306)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:284)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:328)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:356)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:577)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:570)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1538)
      
      2011-01-24 23:23:33,730 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at linux124/10.18.52.124
      ************************************************************/
      
      
      
      1. hdfs-1594.6.patch
        24 kB
        Aaron T. Myers
      2. hdfs-1594.5.patch
        24 kB
        Aaron T. Myers
      3. hdfs-1594.4.patch
        23 kB
        Aaron T. Myers
      4. hdfs-1594.3.patch
        23 kB
        Aaron T. Myers
      5. hdfs-1594.2.patch
        19 kB
        Aaron T. Myers
      6. hdfs-1594.1.patch
        19 kB
        Aaron T. Myers
      7. hdfs-1594.0.patch
        0.8 kB
        Aaron T. Myers
      8. HDFS-1594.patch
        18 kB
        Konstantin Boudnik
      9. HDFS-1594.patch
        17 kB
        Konstantin Boudnik
      10. HDFS-1594.patch
        103 kB
        Devaraj K
      11. hadoop-root-namenode-linux124.log
        36 kB
        Devaraj K

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Devaraj K
            • Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development