Hadoop Common
  1. Hadoop Common
  2. HADOOP-2065

Replication policy for corrupted block

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.1
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Added "corrupt" flag to LocatedBlock to indicate that all replicas of the block thought to be corrupt.

      Description

      Thanks to HADOOP-1955, even if one of the replica is corrupted, the block should get replicated from a good replica relatively fast.

      Created this ticket to continue the discussion from http://issues.apache.org/jira/browse/HADOOP-1955#action_12531162.

      2. Delete corrupted source replica

      3. If all replicas are corrupt, stop replication.

      For (2), it'll be nice if the namenode can delete the corrupted block if there's a good replica on other nodes.

      For (3), I prefer if the namenode can still replicate the block.
      Before 0.14, if the file was corrupted, users were still able to pull the data and decide if they want to delete those files. (HADOOP-2063)
      In 0.14 and later, we cannot/don't replicate these blocks so they eventually get lost.

      To make the matters worse, if the corrupted file is accessed, all the corrupted replicas would be deleted except for one and stay as replication factor of 1 forever.

      1. HADOOP-2065-7.patch
        21 kB
        Lohit Vijayarenu
      2. HADOOP-2065-6.patch
        20 kB
        Lohit Vijayarenu
      3. HADOOP-2065-5.patch
        20 kB
        Lohit Vijayarenu
      4. HADOOP-2065-4.patch
        20 kB
        Lohit Vijayarenu
      5. HADOOP-2065-3.patch
        17 kB
        Lohit Vijayarenu
      6. HADOOP-2065-2.patch
        17 kB
        Lohit Vijayarenu
      7. HADOOP-2065.patch
        9 kB
        Lohit Vijayarenu

        Issue Links

          Activity

          Koji Noguchi created issue -
          Koji Noguchi made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-1955 [ HADOOP-1955 ]
          Koji Noguchi made changes -
          Fix Version/s 0.16.0 [ 12312740 ]
          Nigel Daley made changes -
          Fix Version/s 0.16.0 [ 12312740 ]
          Sameer Paranjpye made changes -
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
          Sameer Paranjpye made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Tsz Wo Nicholas Sze made changes -
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ] lohit vijayarenu [ lohit ]
          Sameer Paranjpye made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Fix Version/s 0.18.0 [ 12312972 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065.patch [ 12380675 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-2.patch [ 12380906 ]
          Lohit Vijayarenu made changes -
          Release Note Changes LocatedBlock by adding a new 'corrupt' flag.
          This patch changes the way replication is handled for corrupt blocks. We retain all corrupt replicas of a block and provide a way flag to check if the block is good or corrupt.
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Incompatible change]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-3.patch [ 12381523 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-4.patch [ 12381616 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-5.patch [ 12381634 ]
          Lohit Vijayarenu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Lohit Vijayarenu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-6.patch [ 12381635 ]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-7.patch [ 12381848 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-7.patch [ 12381848 ]
          Lohit Vijayarenu made changes -
          Attachment HADOOP-2065-7.patch [ 12381849 ]
          Lohit Vijayarenu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Lohit Vijayarenu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Raghu Angadi made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Robert Chansler made changes -
          Release Note Changes LocatedBlock by adding a new 'corrupt' flag.
          This patch changes the way replication is handled for corrupt blocks. We retain all corrupt replicas of a block and provide a way flag to check if the block is good or corrupt.
          Added "corrupt" flag to LocatedBlock to indicate that all replicas of the block thought to be corrupt.
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-4351 [ HADOOP-4351 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]

            People

            • Assignee:
              Lohit Vijayarenu
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development