Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6651

Deletion failure can leak inodes permanently

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      As discussed in HDFS-6618, if a deletion of tree fails in the middle, any collected inodes and blocks will not be removed from INodeMap and BlocksMap.

      Since fsimage is saved by iterating over INodeMap, the leak will persist across name node restart. Although blanked out inodes will not have reference to blocks, blocks will still refer to the inode as BlockCollection. As long as it is not null, blocks will live on. The leaked blocks from blanked out inodes will go away after restart.

      Options (when delete fails in the middle)

      • Complete the partial delete: edit log the partial delete and remove inodes and blocks.
      • Somehow undo the partial delete.
      • Check quota for snapshot diff beforehand for the whole subtree.
      • Ignore quota check during delete even if snapshot is present.
      1. HDFS-6651.002.patch
        82 kB
        Jing Zhao
      2. HDFS-6651.001.patch
        80 kB
        Jing Zhao
      3. HDFS-6651.000.patch
        77 kB
        Jing Zhao

        Activity

        Kihwal Lee created issue -
        Jing Zhao made changes -
        Field Original Value New Value
        Attachment HDFS-6651.000.patch [ 12655128 ]
        Jing Zhao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Jing Zhao [ jingzhao ]
        Jing Zhao made changes -
        Attachment HDFS-6651.001.patch [ 12694886 ]
        Jing Zhao made changes -
        Attachment HDFS-6651.002.patch [ 12695127 ]
        Haohui Mai made changes -
        Summary Deletion failure can leak inodes permanently. Deletion failure can leak inodes permanently
        Haohui Mai made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 2.7.0 [ 12327584 ]
        Resolution Fixed [ 1 ]
        Haohui Mai made changes -
        Hadoop Flags Incompatible change,Reviewed [ 10342, 10343 ]
        Vinod Kumar Vavilapalli made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jing Zhao
            Reporter:
            Kihwal Lee
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development