[HDDS-3816] Erasure Coding

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: OM, Ozone Client, Ozone Datanode, SCM
• Labels: EC pull-request-available
• Target Version/s: 1.3.0
• Assignee: Uma Maheswara Rao G
• Reporter: Uma Maheswara Rao G
• Created: 16/Jun/20 19:59
• Updated: 05/Jan/23 20:11
• Resolved: 20/Apr/22 22:08

Description

We propose to implement Erasure Coding in Apache Ozone to provide efficient storage. With EC in place, Ozone can provide same or better tolerance by giving 50% or more storage space savings.
In HDFS project, we already have native codecs(ISAL) and Java codecs implemented, we can leverage the same or similar codec design.

However, the critical part of EC data layout design is in-progress, we will post the design doc soon.

Also see HDDS-5351, which has a bunch of pre-requisites for the EC feature committed directly to the master branch.

Issue Links

• incorporates: HDDS-6462 Phase II : Erasure Coding Offline Recovery & Read/Write Improvements
• incorporates: HDDS-7731 Phase III: Erasure Coding followup tasks
• is related to: IMPALA-7003 Support erasure-coding in impala
• relates to: HDDS-3755 Storage-class support for Ozone
• requires: HDDS-5351 Erasure Coding Prerequisites
• requires: HDDS-6390 Facilities for compatibility - umbrella jira
• links to: GitHub Pull Request #2868
• links to: GitHub Pull Request #3301
• links to: Upgrade framework integration - problems and possible solutions