Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2251

Namenode does not recognize incorrectly sized blocks

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      We had a lot of file system corruption resulting in incorrectly sized blocks (on disk, they're truncated to 192KB when they should be 64MB).

      However, I cannot make Hadoop realize that these blocks are incorrectly sized. When I try to drain off the node, I get the following messages:

      2008-10-29 18:46:51,293 WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block blk_-4403534125663454855_9937 reported from 172.16.1.150:50010 current size is 67108864 reported size is 196608

      Here 172.16.1.150 is not the node which has the problematic block, but the destination of the file transfer. I propose that Hadoop should either:

      a) Upon startup, make sure that all blocks are properly sized (pro: rather cheap check; con: doesn't catch any truncations which happen while on disk)
      b) Upon detecting the incorrectly sized copy, Hadoop should ask the source of the block to perform a block verification.

      Thanks,

      Brian

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              bockelman Brian Bockelman
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: