Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2065

Replication policy for corrupted block

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.14.1
    • 0.18.0
    • None
    • None
    • Incompatible change
    • Added "corrupt" flag to LocatedBlock to indicate that all replicas of the block thought to be corrupt.

    Description

      Thanks to HADOOP-1955, even if one of the replica is corrupted, the block should get replicated from a good replica relatively fast.

      Created this ticket to continue the discussion from http://issues.apache.org/jira/browse/HADOOP-1955#action_12531162.

      2. Delete corrupted source replica

      3. If all replicas are corrupt, stop replication.

      For (2), it'll be nice if the namenode can delete the corrupted block if there's a good replica on other nodes.

      For (3), I prefer if the namenode can still replicate the block.
      Before 0.14, if the file was corrupted, users were still able to pull the data and decide if they want to delete those files. (HADOOP-2063)
      In 0.14 and later, we cannot/don't replicate these blocks so they eventually get lost.

      To make the matters worse, if the corrupted file is accessed, all the corrupted replicas would be deleted except for one and stay as replication factor of 1 forever.

      Attachments

        1. HADOOP-2065.patch
          9 kB
          Lohit Vijaya Renu
        2. HADOOP-2065-2.patch
          17 kB
          Lohit Vijaya Renu
        3. HADOOP-2065-3.patch
          17 kB
          Lohit Vijaya Renu
        4. HADOOP-2065-4.patch
          20 kB
          Lohit Vijaya Renu
        5. HADOOP-2065-5.patch
          20 kB
          Lohit Vijaya Renu
        6. HADOOP-2065-6.patch
          20 kB
          Lohit Vijaya Renu
        7. HADOOP-2065-7.patch
          21 kB
          Lohit Vijaya Renu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            lohit Lohit Vijaya Renu
            knoguchi Koji Noguchi
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment