Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2882

DN continues to start up, even if block pool fails to initialize

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.2-alpha
    • 2.4.1
    • datanode
    • None
    • Reviewed

    Description

      I started a DN on a machine that was completely out of space on one of its drives. I saw the following:

      2012-02-02 09:56:50,499 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-448349972-172.29.5.192-1323816762969 (storage id DS-507718931-172.29.5.194-11072-12978
      42002148) service to styx01.sf.cloudera.com/172.29.5.192:8021
      java.io.IOException: Mkdirs failed to create /data/1/scratch/todd/styx-datadir/current/BP-448349972-172.29.5.192-1323816762969/tmp
      at org.apache.hadoop.hdfs.server.datanode.FSDataset$BlockPoolSlice.<init>(FSDataset.java:335)

      but the DN continued to run, spewing NPEs when it tried to do block reports, etc. This was on the HDFS-1623 branch but may affect trunk as well.

      Attachments

        1. hdfs-2882.txt
          15 kB
          Todd Lipcon
        2. HDFS-2882.patch
          19 kB
          Vinayakumar B
        3. HDFS-2882.patch
          23 kB
          Vinayakumar B
        4. HDFS-2882.patch
          22 kB
          Vinayakumar B
        5. HDFS-2882.patch
          22 kB
          Vinayakumar B
        6. HDFS-2882.patch
          20 kB
          Vinayakumar B
        7. HDFS-2882.patch
          20 kB
          Vinayakumar B
        8. HDFS-2882.patch
          18 kB
          Vinayakumar B
        9. HDFS-2882.patch
          18 kB
          Vinayakumar B

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vinayakumarb Vinayakumar B Assign to me
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment