Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-381

Datanode should report deletion of blocks to Namenode explicitly

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently datanode notifies namenode newly added blocks and the blocks that are corrupt. There is no explicit message from the datanode to the namenode to indicate the deletion of blocks. Block reports from the datanode is the only way for the namenode to learn about the deletion of blocks at a datanode. With the addition of explicit request to indicate to block deletion, block report interval (which is currently 1 hour) can be increased to a longer duration. This reduces load on both namenode and datanodes.

      1. blockdel.patch
        24 kB
        Suresh Srinivas
      2. blockdel.patch
        24 kB
        Suresh Srinivas
      3. blockdel-1.patch
        3 kB
        Suresh Srinivas
      4. blockdel-2.patch
        4 kB
        Suresh Srinivas
      5. HDFS-381.patch
        4 kB
        Raghu Angadi

        Activity

        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Raghu Angadi made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Raghu Angadi made changes -
        Fix Version/s 0.21.0 [ 12314046 ]
        Raghu Angadi made changes -
        Attachment HDFS-381.patch [ 12411459 ]
        Owen O'Malley made changes -
        Project Hadoop Common [ 12310240 ] HDFS [ 12310942 ]
        Key HADOOP-5724 HDFS-381
        Hadoop Flags [Reviewed, Incompatible change]
        Release Note * When a file is deleted, the corresponding blocks from the blocksMap on Namenode is also deleted
        * Block reports from datanode was a way to indicate deletion of block to the namenode. With this change namenode no longer depends on the block report. Hence reducing the frequency of block reports by increasing the block report interval from current 1 hour to 6 hours.
        Component/s dfs [ 12310710 ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Suresh Srinivas made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Release Note * When a file is deleted, the corresponding blocks from the blocksMap on Namenode is also deleted
        * Block reports from datanode was a way to indicate deletion of block to the namenode. With this change namenode no longer depends on the block report. Hence reducing the frequency of block reports by increasing the block report interval from current 1 hour to 6 hours.
        Raghu Angadi made changes -
        Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Component/s dfs [ 12310710 ]
        Suresh Srinivas made changes -
        Attachment blockdel-2.patch [ 12410734 ]
        Suresh Srinivas made changes -
        Attachment blockdel-1.patch [ 12410235 ]
        Suresh Srinivas made changes -
        Attachment blockdel.patch [ 12406257 ]
        Suresh Srinivas made changes -
        Field Original Value New Value
        Attachment blockdel.patch [ 12406168 ]
        Suresh Srinivas created issue -

          People

          • Assignee:
            Suresh Srinivas
            Reporter:
            Suresh Srinivas
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development