[HDFS-16430] Validate maximum blocks in EC group when adding an EC policy

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.3.1
• Fix Version/s: 3.4.0, 3.2.4, 3.3.5
• Component/s: ec, erasure-coding
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: daimin
• Reporter: daimin
• Due: 18/Jan/22
• Created: 18/Jan/22 07:32
• Updated: 12/Jul/22 10:46
• Resolved: 24/Jan/22 06:35

Description

HDFS EC adopts the last 4 bits of block ID to store the block index in EC block group. Therefore maximum blocks in EC block group is 2^4=16, and which is defined here: HdfsServerConstants#MAX_BLOCKS_IN_GROUP.

Currently there is no limitation or warning when adding a bad EC policy with numDataUnits + numParityUnits > 16. It only results in read/write error on EC file with bad EC policy. To users this is not very straightforward.

Issue Links

• links to: GitHub Pull Request #3899