Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2882

DN continues to start up, even if block pool fails to initialize

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.2-alpha
    • 2.4.1
    • datanode
    • None
    • Reviewed

    Description

      I started a DN on a machine that was completely out of space on one of its drives. I saw the following:

      2012-02-02 09:56:50,499 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-448349972-172.29.5.192-1323816762969 (storage id DS-507718931-172.29.5.194-11072-12978
      42002148) service to styx01.sf.cloudera.com/172.29.5.192:8021
      java.io.IOException: Mkdirs failed to create /data/1/scratch/todd/styx-datadir/current/BP-448349972-172.29.5.192-1323816762969/tmp
      at org.apache.hadoop.hdfs.server.datanode.FSDataset$BlockPoolSlice.<init>(FSDataset.java:335)

      but the DN continued to run, spewing NPEs when it tried to do block reports, etc. This was on the HDFS-1623 branch but may affect trunk as well.

      Attachments

        1. hdfs-2882.txt
          15 kB
          Todd Lipcon
        2. HDFS-2882.patch
          19 kB
          Vinayakumar B
        3. HDFS-2882.patch
          23 kB
          Vinayakumar B
        4. HDFS-2882.patch
          22 kB
          Vinayakumar B
        5. HDFS-2882.patch
          22 kB
          Vinayakumar B
        6. HDFS-2882.patch
          20 kB
          Vinayakumar B
        7. HDFS-2882.patch
          20 kB
          Vinayakumar B
        8. HDFS-2882.patch
          18 kB
          Vinayakumar B
        9. HDFS-2882.patch
          18 kB
          Vinayakumar B

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. HDFS-5529 { Disk Fail } Can we shutdown the DN when it meet's disk failed condition

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-6590 NullPointerException was generated in getBlockLocalPathInfo when datanode restarts

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-4455 Datanode sometimes gives up permanently on Namenode in HA setup

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-4201 NPE in BPServiceActor#sendHeartBeat

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7009 Active NN and standby NN have different live nodes

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-4211 failed volume causes DataNode#getVolumeInfo NPEs on multi-BP DN

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13908 TestDataNodeMultipleRegistrations is flaky

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HDFS-7714 Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.

          • Major - Major loss of function.
          • Closed

          Activity

            People

              vinayakumarb Vinayakumar B
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: