Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-9833

Erasure coding: recomputing block checksum on the fly by reconstructing the missed/corrupt block data

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: None

      Description

      As discussed in HDFS-8430 and HDFS-9694, to compute striped file checksum even some of striped blocks are missed, we need to consider recomputing block checksum on the fly for the missed/corrupt blocks. To recompute the block checksum, the block data needs to be reconstructed by erasure decoding, and the main needed codes for the block reconstruction could be borrowed from HDFS-9719, the refactoring of the existing ErasureCodingWorker. In EC worker, reconstructed blocks need to be written out to target datanodes, but here in this case, the remote writing isn't necessary, as the reconstructed block data is only used to recompute the checksum.

        Attachments

        1. HDFS-9833-00-draft.patch
          47 kB
          Rakesh Radhakrishnan
        2. HDFS-9833-01.patch
          39 kB
          Rakesh Radhakrishnan
        3. HDFS-9833-02.patch
          39 kB
          Rakesh Radhakrishnan
        4. HDFS-9833-03.patch
          41 kB
          Rakesh Radhakrishnan
        5. HDFS-9833-04.patch
          41 kB
          Rakesh Radhakrishnan
        6. HDFS-9833-05.patch
          47 kB
          Rakesh Radhakrishnan
        7. HDFS-9833-06.patch
          47 kB
          Rakesh Radhakrishnan
        8. HDFS-9833-07.patch
          47 kB
          Rakesh Radhakrishnan
        9. HDFS-9833-08.patch
          52 kB
          Rakesh Radhakrishnan

          Issue Links

            Activity

              People

              • Assignee:
                rakeshr Rakesh Radhakrishnan
                Reporter:
                drankye Kai Zheng
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: