[HDFS-4305] Add a configurable limit on number of blocks per file, and min block size

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.4, 2.0.4-alpha
• Fix Version/s: 2.1.0-beta
• Component/s: namenode
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Incompatible change, Reviewed

Release Note

This change introduces a maximum number of blocks per file, by default one million, and a minimum block size, by default 1MB. These can optionally be changed via the configuration settings "dfs.namenode.fs-limits.max-blocks-per-file" and "dfs.namenode.fs-limits.min-block-size", respectively.

• Assignee: Andrew Wang
• Reporter: Todd Lipcon
• Created: 11/Dec/12 21:02
• Updated: 28/Jan/14 17:26
• Resolved: 29/Apr/13 21:58

Description

We recently had an issue where a user set the block size very very low and managed to create a single file with hundreds of thousands of blocks. This caused problems with the edit log since the OP_ADD op was so large (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To prevent users from making such mistakes, we should:

• introduce a configurable minimum block size, below which requests are rejected
• introduce a configurable maximum number of blocks per file, above which requests to add another block are rejected (with a suitably high default as to not prevent legitimate large files)

Issue Links

• is duplicated by: HDFS-5463 NameNode should limit the number of blocks per file
• is related to: ACCUMULO-2266 TServer should ensure wal settings are valid for underlying FS