[HDFS-7337] Configurable and pluggable erasure codec and policy

• Type: New Feature
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta1
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-nice-to-have
• Target Version/s: 3.0.0-beta1
• Hadoop Flags: Incompatible change

Release Note

This allows users to:
* develop and plugin their own erasure codec and coders. The plugin will be loaded automatically from hadoop jars, the corresponding codec and coder will be registered for runtime use.
* define their own erasure coding policies thru an xml file and CLI command. The added policies will be persisted into fsimage.

• Assignee: Sammi Chen
• Reporter: Zhe Zhang
• Created: 04/Nov/14 01:22
• Updated: 02/Oct/19 17:14
• Resolved: 21/Sep/17 07:01

Description

According to HDFS-7285 and the design, this considers to support multiple Erasure Codecs via pluggable approach. It allows to define and configure multiple codec schemas with different coding algorithms and parameters. The resultant codec schemas can be utilized and specified via command tool for different file folders. While design and implement such pluggable framework, it’s also to implement a concrete codec by default (Reed Solomon) to prove the framework is useful and workable. Separate JIRA could be opened for the RS codec implementation.

Note HDFS-7353 will focus on the very low level codec API and implementation to make concrete vendor libraries transparent to the upper layer. This JIRA focuses on high level stuffs that interact with configuration, schema and etc.

Issue Links

• contains: HADOOP-11649 Erasure Coding: Load multiple erasure codecs
• contains: HADOOP-13200 Implement customizable and configurable erasure coders
• incorporates: HADOOP-11514 Raw Erasure Coder API for concrete encoding and decoding
• incorporates: HADOOP-11645 Erasure Codec API covering the essential aspects for an erasure code
• incorporates: HADOOP-11646 Erasure Coder API for encoding and decoding of block group
• incorporates: HADOOP-11664 Loading predefined EC schemas from configuration
• incorporates: HDFS-7363 Pluggable algorithms to form block groups in erasure coding
• is depended upon by: HDFS-7345 Local Repairable Codes (LRC)
• is related to: HDFS-8031 Follow-on work for erasure coding phase I (striping layout)

Sub-Tasks

• Erasure Coding: perform stripping erasure encoding work given block reader and writer: Resolved
• Erasure Coding: createErasureCodingZone api should accept the policyname as argument instead of ErasureCodingPolicy: Resolved
• Define and parse erasure code policies: Resolved
• Allow user to customize new erasure code policies: Resolved
• Add CLI cmd to remove an erasure code policy: Resolved
• Add ec sub command -listCodec to show currently supported ec codecs: Resolved
• Provide a system-default EC policy: Resolved