[HADOOP-11264] Common side changes for HDFS Erasure coding support

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HDFS-7285
• Component/s: io
• Hadoop Flags: Reviewed
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 04/Nov/14 12:44
• Updated: 21/Apr/20 08:39
• Resolved: 21/Apr/20 08:39

Description

This is umbrella JIRA for tracking the common side changes for HDFS Erasure Coding support.

Issue Links

• is part of: HDFS-7285 Erasure Coding Support inside HDFS
• is related to: HADOOP-11842 Common side change for follow-on work for erasure coding phase I
• relates to: HADOOP-12633 Extend Erasure Code to support POWER Chip acceleration

Sub-Tasks

• Raw Erasure Coder API for concrete encoding and decoding: Resolved
• Minor improvements for raw erasure coders: Resolved
• Raw Reed-Solomon coder using Intel ISA-L library: Resolved
• Raw XOR coder: Resolved
• Raw Reed-Solomon coder in pure Java: Resolved
• Define EC schema API for ErasureCodec: Resolved
• Erasure Codec API covering the essential aspects for an erasure code: Resolved
• Erasure Coder API for encoding and decoding of block group: Resolved
• Reed-Solomon ErasureCoder: Resolved
• Loading predefined EC schemas from configuration: Resolved
• Make erasure coder configurable: Resolved
• Add factory to create raw erasure coder: Resolved
• Add API to NetworkTopology for getting all racks: Resolved
• Refine a little bit erasure coder API: Resolved
• Combine erasure encoder and decoder interfaces: Resolved
• Correct two thrown messages in ECSchema class: Resolved
• Better to rename some raw erasure coders: Resolved
• Minor improvements for erasurecode classes: Resolved
• ECSchema are considered as not equal even if they are logically equal: Open
• Remove ecschema-def.xml: Resolved
• Enhance raw coder allowing to read least required inputs in decoding: Resolved
• Enhance erasure coder allowing to read least required inputs in decoding of block group: Open
• Introduce Intel ISA-L erasure coding library for native erasure encoding support: Resolved
• Erasure coding: Should be able to encode part of parity blocks.: Resolved
• Add tests and fix for erasure coders to recover erased parity units: Resolved
• Refactor some codes for erasure coders: Resolved
• Enhance tests for erasure coders: Resolved
• Enhance ByteBuffer version encode/decode API of raw erasure coder: Resolved
• Add interface of whether codec has chunk boundary to Erasure coder: Resolved
• Improve and restructure native ISAL support: Resolved
• Remove chunkSize from ECSchema as its not required for coders: Resolved
• Address some issues related to ByteBuffer type input/output buffers for raw erasure coders: Resolved
• Using more meaningful keys in EC schema: Resolved