[HDFS-6102] Lower the default maximum items per directory to fix PB fsimage loading

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: namenode
• Hadoop Flags: Incompatible change
• Assignee: Andrew Wang
• Reporter: Andrew Wang
• Created: 13/Mar/14 18:27
• Updated: 08/Aug/16 02:37
• Resolved: 14/Mar/14 05:01

Description

Found by Stephen Chu during testing. We were creating a bunch of directories in a single directory to blow up the fsimage size, and it ends up we hit this error when trying to load a very large fsimage:

2014-03-13 13:57:03,901 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode: Loading 24523605 INodes.
2014-03-13 13:57:59,038 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage: Failed to load image from FSImageFile(file=/dfs/nn/current/fsimage_0000000000024532742, cpktTxId=0000000000024532742)
com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
        at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
        at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
        at com.google.protobuf.CodedInputStream.readRawByte(CodedInputStream.java:769)
        at com.google.protobuf.CodedInputStream.readRawVarint64(CodedInputStream.java:462)
        at com.google.protobuf.CodedInputStream.readUInt64(CodedInputStream.java:188)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry.<init>(FsImageProto.java:9839)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry.<init>(FsImageProto.java:9770)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry$1.parsePartialFrom(FsImageProto.java:9901)
        at org.apache.hadoop.hdfs.server.namenode.FsImageProto$INodeDirectorySection$DirEntry$1.parsePartialFrom(FsImageProto.java:9896)
        at 52)
...

Some further research reveals there's a 64MB max size per PB message, which seems to be what we're hitting here.

Issue Links

• is related to: HDFS-6104 TestFsLimits#testDefaultMaxComponentLength Fails on branch-2
• is related to: HDFS-10729 Improve log message for edit loading failures caused by FS limit checks.
• is related to: HDFS-7473 Document setting dfs.namenode.fs-limits.max-directory-items to 0 is invalid
• is related to: HDFS-10593 MAX_DIR_ITEMS should not be hard coded since RPC buff size is configurable
• relates to: HDFS-6099 HDFS file system limits not enforced on renames.
• relates to: HDFS-6055 Change default configuration to limit file name length in HDFS