Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1594

When the disk becomes full Namenode is getting shutdown and not able to recover

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.21.1, 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Environment:

      Linux linux124 2.6.27.19-5-default #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.
      Show
      Implemented a daemon thread to monitor the disk usage for periodically and if the disk usage reaches the threshold value, put the name node into Safe mode so that no modification to file system will occur. Once the disk usage reaches below the threshold, name node will be put out of the safe mode. Here threshold value and interval to check the disk usage are configurable.

      Description

      When the disk becomes full name node is shutting down and if we try to start after making the space available It is not starting and throwing the below exception.

       
      
      2011-01-24 23:23:33,727 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
      java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:180)
      	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:117)
      	at org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.readString(FSImageSerialization.java:201)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:60)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1089)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1041)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:487)
      	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:149)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:306)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:284)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:328)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:356)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:577)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:570)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1538)
      2011-01-24 23:23:33,729 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:180)
      	at org.apache.hadoop.io.UTF8.readFields(UTF8.java:117)
      	at org.apache.hadoop.hdfs.server.namenode.FSImageSerialization.readString(FSImageSerialization.java:201)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:185)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:93)
      	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:60)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1089)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1041)
      	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:487)
      	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:149)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:306)
      	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:284)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:328)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:356)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:577)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:570)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1529)
      	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1538)
      
      2011-01-24 23:23:33,730 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
      /************************************************************
      SHUTDOWN_MSG: Shutting down NameNode at linux124/10.18.52.124
      ************************************************************/
      
      
      

        Attachments

        1. hadoop-root-namenode-linux124.log
          36 kB
          Devaraj K
        2. HDFS-1594.patch
          103 kB
          Devaraj K
        3. HDFS-1594.patch
          17 kB
          Konstantin Boudnik
        4. HDFS-1594.patch
          18 kB
          Konstantin Boudnik
        5. hdfs-1594.0.patch
          0.8 kB
          Aaron T. Myers
        6. hdfs-1594.1.patch
          19 kB
          Aaron T. Myers
        7. hdfs-1594.2.patch
          19 kB
          Aaron T. Myers
        8. hdfs-1594.3.patch
          23 kB
          Aaron T. Myers
        9. hdfs-1594.4.patch
          23 kB
          Aaron T. Myers
        10. hdfs-1594.5.patch
          24 kB
          Aaron T. Myers
        11. hdfs-1594.6.patch
          24 kB
          Aaron T. Myers

          Issue Links

            Activity

              People

              • Assignee:
                atm Aaron T. Myers
                Reporter:
                devaraj.k Devaraj K
              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: