[HDFS-2006] ability to support storing extended attributes per file

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HDFS XAttrs (HDFS-2006)
• Fix Version/s: 2.5.0
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Yi Liu
• Reporter: Dhruba Borthakur
• Created: 27/May/11 04:26
• Updated: 03/Sep/14 20:34
• Resolved: 11/Jun/14 19:33

Description

It would be nice if HDFS provides a feature to store extended attributes for files, similar to the one described here: http://en.wikipedia.org/wiki/Extended_file_attributes.
The challenge is that it has to be done in such a way that a site not using this feature does not waste precious memory resources in the namenode.

Issue Links

• incorporates: HADOOP-10514 Common side changes to support HDFS extended attributes (HDFS-2006)
• is related to: HDFS-6556 Refine XAttr permissions
• is related to: HADOOP-7269 S3 Native should allow customizable file meta-data (headers)
• is required by: HADOOP-10150 Hadoop cryptographic file system
• relates to: HDFS-6395 Skip checking xattr limits for non-user-visible namespaces
• relates to: HDFS-6464 Support multiple xattr.name parameters for WebHDFS getXAttrs.
• relates to: MAPREDUCE-5898 distcp to support preserving HDFS extended attributes(XAttrs)
• relates to: MAPREDUCE-5920 Add Xattr option in DistCp docs
• relates to: HDFS-4672 Support tiered storage policies
• relates to: HDFS-6260 Support extended attributes via libHDFS
• relates to: HDFS-6375 Listing extended attributes with the search permission
• relates to: HDFS-6430 HTTPFS - Implement XAttr support
• relates to: HDFS-6486 Add user doc for XAttrs via WebHDFS.

Sub-Tasks

• Protobuf for XAttr and client-side implementation: Resolved
• Implement XAttr as a INode feature.: Resolved
• Namenode server-side storage for XAttrs: Resolved
• HDFS implementation of FileContext API for XAttrs.: Resolved
• Shift XAttr helper code out for reuse.: Resolved
• NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log.: Resolved
• XML based End-to-End test for getfattr and setfattr commands: Resolved
• Test cases for XAttrs: Resolved
• Javadocs for Xattrs apis in DFSClient and other minor fixups: Resolved
• ClientProtocol#setXattr should not be annotated idempotent: Resolved
• TestOfflineEditsViewer for XAttr: Resolved
• Maximum limit on the size of an xattr: Resolved
• Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call: Resolved
• fix TestNamenodeRetryCache and TestRetryCacheWithHA failures: Resolved
• SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA: Resolved
• FsImage loading failed with RemoveXattr op: Resolved
• Clean up audit logging in FSNamesystem xattr methods: Resolved
• Write end user documentation for xattrs.: Resolved
• Handle setXattr rpcIDs for OfflineEditsViewer: Resolved
• Remove support for extended attributes on symlinks: Resolved
• Unify xattr name and value limits into a single limit: Resolved
• Support extended attributes via WebHDFS: Resolved
• setXAttr should require the user to be the owner of the file or directory: Resolved
• Should have different size limits for different XAttr namespaces (user, trusted, security and system).: Resolved
• xattr names erroneously handled as case-insensitive.: Resolved
• xattr modification operations are based on state of latest snapshot instead of current version of inode.: Resolved
• DFSClient unwraps AclException in xattr methods, but those methods cannot throw AclException.: Resolved
• Interface audience and stability annotations missing from several new classes related to xattrs.: Resolved