[HDFS-7285] Erasure Coding Support inside HDFS

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha1
• Hadoop Flags: Reviewed

Release Note

<!-- markdown -->
HDFS now provides native support for erasure coding (EC) to store data more efficiently. Each individual directory can be configured with an EC policy with command `hdfs erasurecode -setPolicy`. When a file is created, it will inherit the EC policy from its nearest ancestor directory to determine how its blocks are stored. Compared to 3-way replication, the default EC policy saves 50% of storage space while also tolerating more storage failures.

To support small files, the currently phase of HDFS-EC stores blocks in _striped_ layout, where a logical file block is divided into small units (64KB by default) and distributed to a set of DataNodes. This enables parallel I/O but also decreases data locality. Therefore, the cluster environment and I/O workloads should be considered before configuring EC policies.

• Assignee: Zhe Zhang
• Reporter: Weihua Jiang
• Created: 24/Oct/14 13:02
• Updated: 29/Apr/21 06:20
• Resolved: 30/Sep/15 15:52

Description

Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data.

Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back.

We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon.

Issue Links

• incorporates: HADOOP-11264 Common side changes for HDFS Erasure coding support
• is depended upon by: HDFS-8030 HDFS Erasure Coding Phase II -- EC with contiguous layout
• is depended upon by: HDFS-8031 Follow-on work for erasure coding phase I (striping layout)
• is related to: HBASE-19954 Separate TestBlockReorder into individual tests to avoid ShutdownHook suppression error against hadoop3
• is related to: HDFS-503 Implement erasure coding as a layer on HDFS
• is related to: HDFS-2832 Enable support for heterogeneous storages in HDFS - DN as a collection of storages
• is related to: HDFS-6584 Support Archival Storage
• is related to: HDFS-7343 HDFS smart storage management
• relates to: HADOOP-12633 Extend Erasure Code to support POWER Chip acceleration
• supercedes: MAPREDUCE-3868 Reenable Raid