Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7337

Configurable and pluggable erasure codec and policy

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      This allows users to:
      * develop and plugin their own erasure codec and coders. The plugin will be loaded automatically from hadoop jars, the corresponding codec and coder will be registered for runtime use.
      * define their own erasure coding policies thru an xml file and CLI command. The added policies will be persisted into fsimage.
      Show
      This allows users to: * develop and plugin their own erasure codec and coders. The plugin will be loaded automatically from hadoop jars, the corresponding codec and coder will be registered for runtime use. * define their own erasure coding policies thru an xml file and CLI command. The added policies will be persisted into fsimage.

      Description

      According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via pluggable approach. It allows to define and configure multiple codec schemas with different coding algorithms and parameters. The resultant codec schemas can be utilized and specified via command tool for different file folders. While design and implement such pluggable framework, it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework is useful and workable. Separate JIRA could be opened for the RS codec implementation.

      Note HDFS-7353 will focus on the very low level codec API and implementation to make concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level stuffs that interact with configuration, schema and etc.

        Attachments

        1. PluggableErasureCodec v4.pdf
          154 kB
          Sammi Chen
        2. PluggableErasureCodec-v3.pdf
          363 kB
          Kai Zheng
        3. PluggableErasureCodec-v2.pdf
          432 kB
          Kai Zheng
        4. HDFS-7337-prototype-v3.zip
          49 kB
          Kai Zheng
        5. HDFS-7337-prototype-v2.zip
          34 kB
          Kai Zheng
        6. PluggableErasureCodec.pdf
          409 kB
          Kai Zheng
        7. HDFS-7337-prototype-v1.patch
          14 kB
          Kai Zheng

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                Sammi Sammi Chen
                Reporter:
                zhz Zhe Zhang
              • Votes:
                2 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: