[HDFS-6055] Change default configuration to limit file name length in HDFS

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0, 3.0.0-alpha1
• Fix Version/s: 2.4.0
• Component/s: namenode
• Target Version/s: 2.4.0
• Hadoop Flags: Incompatible change, Reviewed

Release Note

The default configuration of HDFS now sets dfs.namenode.fs-limits.max-component-length to 255 for improved interoperability with other file system implementations. This limits each component of a file system path to a maximum of 255 bytes in UTF-8 encoding. Attempts to create new files that violate this rule will fail with an error. Existing files that violate the rule are not effected. Previously, dfs.namenode.fs-limits.max-component-length was set to 0 (ignored). If necessary, it is possible to set the value back to 0 in the cluster's configuration to restore the old behavior.

• Assignee: Chris Nauroth
• Reporter: Suresh Srinivas
• Created: 05/Mar/14 00:02
• Updated: 12/May/16 18:15
• Resolved: 10/Mar/14 22:00

Description

Currently configuration "dfs.namenode.fs-limits.max-component-length" is set to 0. With this HDFS file names have no length limit. However, we see more users run into issues where they copy files from HDFS to another file system and the copy fails due to the file name length being too long.

I propose changing the default configuration "dfs.namenode.fs-limits.max-component-length" to a reasonable value. This will be an incompatible change. However, user who need long file names can override this configuration to turn off length limit.

What do folks think?

Issue Links

• is related to: HDFS-6102 Lower the default maximum items per directory to fix PB fsimage loading
• is related to: HDFS-6099 HDFS file system limits not enforced on renames.
• is related to: HDFS-240 Should HDFS restrict the names used for files?
• relates to: HDFS-1751 Intrinsic limits for HDFS files, directories