[HDFS-6055] Change default configuration to limit file name length in HDFS - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.4.0, 3.0.0-alpha1
Fix Version/s: 2.4.0
Component/s: namenode
Labels:
None

Target Version/s:

2.4.0
Hadoop Flags:

Incompatible change, Reviewed
Release Note:

Hide
The default configuration of HDFS now sets dfs.namenode.fs-limits.max-component-length to 255 for improved interoperability with other file system implementations. This limits each component of a file system path to a maximum of 255 bytes in UTF-8 encoding. Attempts to create new files that violate this rule will fail with an error. Existing files that violate the rule are not effected. Previously, dfs.namenode.fs-limits.max-component-length was set to 0 (ignored). If necessary, it is possible to set the value back to 0 in the cluster's configuration to restore the old behavior.

Show
The default configuration of HDFS now sets dfs.namenode.fs-limits.max-component-length to 255 for improved interoperability with other file system implementations. This limits each component of a file system path to a maximum of 255 bytes in UTF-8 encoding. Attempts to create new files that violate this rule will fail with an error. Existing files that violate the rule are not effected. Previously, dfs.namenode.fs-limits.max-component-length was set to 0 (ignored). If necessary, it is possible to set the value back to 0 in the cluster's configuration to restore the old behavior.

Description

Currently configuration "dfs.namenode.fs-limits.max-component-length" is set to 0. With this HDFS file names have no length limit. However, we see more users run into issues where they copy files from HDFS to another file system and the copy fails due to the file name length being too long.

I propose changing the default configuration "dfs.namenode.fs-limits.max-component-length" to a reasonable value. This will be an incompatible change. However, user who need long file names can override this configuration to turn off length limit.

What do folks think?