Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6833

DirectoryScanner should not register a deleting block with memory of DataNode

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5.0, 2.5.1, 3.0.0-alpha1
    • Fix Version/s: 2.7.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When a block is deleted in DataNode, the following messages are usually output.

      2014-08-07 17:53:11,606 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825 for deletion
      2014-08-07 17:53:11,617 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
      

      However, DirectoryScanner may be executed when DataNode deletes the block in the current implementation. And the following messsages are output.

      2014-08-07 17:53:30,519 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825 for deletion
      2014-08-07 17:53:31,426 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata files:0, missing block files:0, missing blocks in memory:1, mismatched blocks:0
      2014-08-07 17:53:31,426 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
        getNumBytes()     = 21230663
        getBytesOnDisk()  = 21230663
        getVisibleLength()= 21230663
        getVolume()       = /hadoop/data1/dfs/data/current
        getBlockFile()    = /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
        unlinked          =false
      2014-08-07 17:53:31,531 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
      

      Deleting block information is registered in DataNode's memory.
      And when DataNode sends a block report, NameNode receives wrong block information.

      For example, when we execute recommission or change the number of replication, NameNode may delete the right block as "ExcessReplicate" by this problem.
      And "Under-Replicated Blocks" and "Missing Blocks" occur.

      When DataNode run DirectoryScanner, DataNode should not register a deleting block.

        Attachments

        1. HDFS-6833-16.patch
          14 kB
          Shinichi Yamashita
        2. HDFS-6833-15.patch
          15 kB
          Shinichi Yamashita
        3. HDFS-6833-14.patch
          14 kB
          Shinichi Yamashita
        4. HDFS-6833-13.patch
          13 kB
          Shinichi Yamashita
        5. HDFS-6833-12.patch
          14 kB
          Shinichi Yamashita
        6. HDFS-6833-11.patch
          13 kB
          Shinichi Yamashita
        7. HDFS-6833-10.patch
          17 kB
          Shinichi Yamashita
        8. HDFS-6833.9.patch
          13 kB
          Shinichi Yamashita
        9. HDFS-6833.8.patch
          16 kB
          Tsuyoshi Ozawa
        10. HDFS-6833-6-3.patch
          13 kB
          Shinichi Yamashita
        11. HDFS-6833-6-2.patch
          13 kB
          Shinichi Yamashita
        12. HDFS-6833-7-2.patch
          16 kB
          Shinichi Yamashita
        13. HDFS-6833-7.patch
          16 kB
          Shinichi Yamashita
        14. HDFS-6833-6.patch
          13 kB
          Shinichi Yamashita
        15. HDFS-6833.patch
          11 kB
          Shinichi Yamashita
        16. HDFS-6833.patch
          11 kB
          Shinichi Yamashita
        17. HDFS-6833.patch
          11 kB
          Shinichi Yamashita
        18. HDFS-6833.patch
          7 kB
          Shinichi Yamashita
        19. HDFS-6833.patch
          7 kB
          Shinichi Yamashita

          Activity

            People

            • Assignee:
              sinchii Shinichi Yamashita
              Reporter:
              sinchii Shinichi Yamashita
            • Votes:
              1 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: