Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-9833

Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0-alpha1
    • None

    Description

      As discussed in HDFS-8430 and HDFS-9694, to compute striped file checksum even some of striped blocks are missed, we need to consider recomputing block checksum on the fly for the missed/corrupt blocks. To recompute the block checksum, the block data needs to be reconstructed by erasure decoding, and the main needed codes for the block reconstruction could be borrowed from HDFS-9719, the refactoring of the existing ErasureCodingWorker. In EC worker, reconstructed blocks need to be written out to target datanodes, but here in this case, the remote writing isn't necessary, as the reconstructed block data is only used to recompute the checksum.

      Attachments

        1. HDFS-9833-00-draft.patch
          47 kB
          Rakesh Radhakrishnan
        2. HDFS-9833-01.patch
          39 kB
          Rakesh Radhakrishnan
        3. HDFS-9833-02.patch
          39 kB
          Rakesh Radhakrishnan
        4. HDFS-9833-03.patch
          41 kB
          Rakesh Radhakrishnan
        5. HDFS-9833-04.patch
          41 kB
          Rakesh Radhakrishnan
        6. HDFS-9833-05.patch
          47 kB
          Rakesh Radhakrishnan
        7. HDFS-9833-06.patch
          47 kB
          Rakesh Radhakrishnan
        8. HDFS-9833-07.patch
          47 kB
          Rakesh Radhakrishnan
        9. HDFS-9833-08.patch
          52 kB
          Rakesh Radhakrishnan

        Issue Links

          Activity

            People

              rakeshr Rakesh Radhakrishnan
              drankye Kai Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: