Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2012

Periodic verification at the Datanode

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.16.0
    • None
    • None

    Description

      Currently on-disk data corruption on data blocks is detected only when it is read by the client or by another datanode. These errors are detected much earlier if datanode can periodically verify the data checksums for the local blocks.

      Some of the issues to consider :

      • How should we check the blocks ( no more often than once every couple of weeks ?)
      • How do we keep track of when a block was last verfied ( there is a .meta file associcated with each lock ).
      • What action to take once a corruption is detected
      • Scanning should be done as a very low priority with rest of the datanode disk traffic in mind.

      Attachments

        1. HADOOP-2012.patch
          61 kB
          Raghu Angadi
        2. HADOOP-2012.patch
          62 kB
          Raghu Angadi
        3. HADOOP-2012.patch
          61 kB
          Raghu Angadi
        4. HADOOP-2012.patch
          26 kB
          Raghu Angadi
        5. HADOOP-2012.patch
          61 kB
          Raghu Angadi
        6. HADOOP-2012.patch
          61 kB
          Raghu Angadi
        7. HADOOP-2012.patch
          56 kB
          Raghu Angadi
        8. HADOOP-2012.patch
          51 kB
          Raghu Angadi
        9. HADOOP-2012.patch
          49 kB
          Raghu Angadi
        10. HADOOP-2012.patch
          69 kB
          Raghu Angadi
        11. HADOOP-2012.patch
          22 kB
          Raghu Angadi
        12. HADOOP-2012.patch
          22 kB
          Raghu Angadi
        13. HADOOP-2012.patch
          22 kB
          Raghu Angadi
        14. HADOOP-2012.patch
          21 kB
          Raghu Angadi

        Issue Links

          Activity

            People

              rangadi Raghu Angadi
              rangadi Raghu Angadi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: