[HDFS-8030] HDFS Erasure Coding Phase II -- EC with contiguous layout

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: erasure-coding
• Assignee: Zhe Zhang
• Reporter: Zhe Zhang
• Created: 31/Mar/15 17:20
• Updated: 19/Mar/16 00:40

Description

Data redundancy form – replication or erasure coding, should be orthogonal to block layout – contiguous or striped. This JIRA explores the combination of Erasure Coding + Contiguous block layout.

As will be detailed in the design document, key benefits include preserving block locality, and easy conversion between hot and cold modes.

Issue Links

• depends upon: HDFS-7285 Erasure Coding Support inside HDFS
• relates to: HDFS-8031 Follow-on work for erasure coding phase I (striping layout)

Sub-Tasks

• Erasure Coding: perform non-stripping erasure encoding work given block reader and writer: Open
• Erasure Coding: perform non-stripping erasure decoding/recovery work given block reader and writer: Open
• Erasure Coding: handle read/write failure for non-stripping coding blocks: Open
• Erasure Coding: restore BlockGroup and schema info from non-stripping coding command: Open
• Erasure Coding: on-demand erasure recovery for erased data serving client request in DataNode: Open
• Erasure coding: distribute recovery work for non-stripping erasure coding blocks to DataNode: Open
• Erasure Coding: distribute replication to non-stripping erasure coding conversion work to DataNode: Open
• Erasure Coding: DN should report its CPU cores and other info to NN for better EC task assignments: Open
• Use block erasure coder in client stripping: Reopened