Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6185 HDFS operational and debuggability improvements
  3. HDFS-6186

Pause deletion of blocks when the namenode starts up

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.5.0
    • namenode
    • None
    • Reviewed

    Description

      HDFS namenode can delete blocks very quickly, given the deletion happens as a parallel operation spread across many datanodes. One of the frequent anxieties I see is that a lot of data can be deleted very quickly, when a cluster is brought up, especially when one of the storage directories has failed and namenode metadata was copied from another storage. Copying wrong metadata would results in some of the newer files (if old metadata was copied) being deleted along with their blocks.

      HDFS-5986 now captures the number of pending deletion block on namenode webUI and JMX. I propose pausing deletion of blocks for a configured period of time (default 1 hour?) after namenode comes out of safemode. This will give enough time for the administrator to notice large number of pending deletion blocks and take corrective action.

      Thoughts?

      Attachments

        1. HDFS-6186.000.patch
          12 kB
          Jing Zhao
        2. HDFS-6186.002.patch
          11 kB
          Jing Zhao
        3. HDFS-6186.003.patch
          12 kB
          Jing Zhao

        Issue Links

          Activity

            People

              jingzhao Jing Zhao
              sureshms Suresh Srinivas
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: