[HDFS-7339] Representing striped block groups in NameNode with hierarchical naming protocol

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Target Version/s: HDFS-7285
• Hadoop Flags: Reviewed
• Assignee: Zhe Zhang
• Reporter: Zhe Zhang
• Created: 04/Nov/14 01:25
• Updated: 30/Sep/15 19:37
• Resolved: 31/Jan/15 00:21

Description

All erasure codec operations center around the concept of block group; they are formed in initial encoding and looked up in recoveries and conversions. A lightweight class BlockGroup is created to record the original and parity blocks in a coding group, as well as a pointer to the codec schema (pluggable codec schemas will be supported in HDFS-7337). With the striping layout, the HDFS client needs to operate on all blocks in a BlockGroup concurrently. Therefore we propose to extend a file’s inode to switch between contiguous and striping modes, with the current mode recorded in a binary flag. An array of BlockGroups (or BlockGroup IDs) is added, which remains empty for “traditional” HDFS files with contiguous block layout.

The NameNode creates and maintains BlockGroup instances through the new ECManager component; the attached figure has an illustration of the architecture. As a simple example, when a {Striping+EC} file is created and written to, it will serve requests from the client to allocate new BlockGroups and store them under the INodeFile. In the current phase, BlockGroups are allocated both in initial online encoding and in the conversion from replication to EC. ECManager also facilitates the lookup of BlockGroup information for block recovery work.