Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
  3. HDFS-7344

[umbrella] Erasure Coding worker and support in DataNode

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Done
    • None
    • None
    • datanode
    • None

    Description

      According to HDFS-7285 and the design, this handles DataNode side extension and related support for Erasure Coding. More specifically, it implements ECWorker, which reconstructs lost blocks (in striping layout).

      It generally needs to restore BlockGroup and schema information from coding commands from NameNode or other entities, and construct specific coding work to execute. The required block reader, writer, either local or remote, encoder and decoder, will be implemented separately as sub-tasks.

      This JIRA will track all the linked sub-tasks, and is responsible for general discussions and integration for ECWorker. It won't resolve until all the related tasks are done.

      Attachments

        1. ECWorker-design-v2.pdf
          430 kB
          Kai Zheng
        2. hdfs-ec-datanode.0108.zip
          37 kB
          Li Bo
        3. hdfs-ec-datanode.0108.zip
          37 kB
          skyfly_99
        4. HDFS ECWorker Design.pdf
          368 kB
          Li Bo

        Issue Links

          contains

          Sub-task - The sub-task of the issue HDFS-8019 Erasure Coding: erasure coding chunk buffer allocation and management

          • Major - Major loss of function.
          • In Progress

          Sub-task - The sub-task of the issue HDFS-7348 Erasure Coding: DataNode reconstruct striped blocks

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-8014 Erasure Coding: local and remote block reader for coding work in DataNode

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-8015 Erasure Coding: local and remote block writer for coding work in DataNode

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-8024 Erasure Coding: ECworker frame, basics, bootstraping and configuration

          • Major - Major loss of function.
          • Resolved

          Task - A task that needs to be done. HDFS-8020 Erasure Coding: constructing and mapping BlockGroup info to utilize and call ErasureCoder API

          • Major - Major loss of function.
          • Patch Available
          is related to

          Sub-task - The sub-task of the issue HDFS-7717 Erasure Coding: distribute replication to EC conversion work to DataNode

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HDFS-8013 Erasure coding: distribute recovery work for non-stripping erasure coding blocks to DataNode

          • Major - Major loss of function.
          • Open

          Sub-task - The sub-task of the issue HDFS-7369 Erasure coding: distribute recovery work for striped blocks to DataNode

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-7679 Erasure Coding: unifying common constructs like coding work, block reader and block writer across client and DataNode

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HDFS-8023 Erasure Coding: retrieve eraure coding schema for a file from NameNode

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              libo-intel Li Bo
              drankye Kai Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: