Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2006

ability to support storing extended attributes per file

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HDFS XAttrs (HDFS-2006)
    • Fix Version/s: 2.5.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be nice if HDFS provides a feature to store extended attributes for files, similar to the one described here: http://en.wikipedia.org/wiki/Extended_file_attributes.
      The challenge is that it has to be done in such a way that a site not using this feature does not waste precious memory resources in the namenode.

      1. xattrs.patch
        129 kB
        Yi Liu
      2. HDFS-XAttrs-Design-1.pdf
        402 kB
        Yi Liu
      3. xattrs.1.patch
        138 kB
        Yi Liu
      4. HDFS-XAttrs-Design-2.pdf
        424 kB
        Yi Liu
      5. HDFS-XAttrs-Design-3.pdf
        434 kB
        Yi Liu
      6. Test-Plan-for-Extended-Attributes-1.pdf
        305 kB
        Yi Liu
      7. ExtendedAttributes.html
        22 kB
        Uma Maheswara Rao G
      8. HDFS-2006-Merge-1.patch
        258 kB
        Uma Maheswara Rao G
      9. HDFS-2006-Merge-2.patch
        266 kB
        Uma Maheswara Rao G
      10. HDFS-2006-Branch-2-Merge.patch
        278 kB
        Uma Maheswara Rao G

        Issue Links

        1.
        Protobuf for XAttr and client-side implementation Sub-task Resolved Yi Liu
         
        2.
        Implement XAttr as a INode feature. Sub-task Resolved Yi Liu
         
        3.
        Namenode server-side storage for XAttrs Sub-task Resolved Yi Liu
         
        4.
        HDFS implementation of FileContext API for XAttrs. Sub-task Resolved Yi Liu
         
        5.
        Shift XAttr helper code out for reuse. Sub-task Resolved Yi Liu
         
        6.
        NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log. Sub-task Resolved Yi Liu
         
        7.
        XML based End-to-End test for getfattr and setfattr commands Sub-task Resolved Yi Liu
         
        8.
        Test cases for XAttrs Sub-task Resolved Yi Liu
         
        9.
        Javadocs for Xattrs apis in DFSClient and other minor fixups Sub-task Resolved Charles Lamb
         
        10.
        ClientProtocol#setXattr should not be annotated idempotent Sub-task Resolved Uma Maheswara Rao G
         
        11.
        TestOfflineEditsViewer for XAttr Sub-task Resolved Yi Liu
         
        12.
        Maximum limit on the size of an xattr Sub-task Resolved Yi Liu
         
        13.
        Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call Sub-task Resolved Yi Liu
         
        14.
        fix TestNamenodeRetryCache and TestRetryCacheWithHA failures Sub-task Resolved Uma Maheswara Rao G
         
        15.
        SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA Sub-task Resolved Uma Maheswara Rao G
         
        16.
        FsImage loading failed with RemoveXattr op Sub-task Resolved Uma Maheswara Rao G
         
        17.
        Clean up audit logging in FSNamesystem xattr methods Sub-task Resolved Andrew Wang
         
        18.
        Write end user documentation for xattrs. Sub-task Resolved Andrew Wang
         
        19.
        Handle setXattr rpcIDs for OfflineEditsViewer Sub-task Resolved Uma Maheswara Rao G
         
        20.
        Remove support for extended attributes on symlinks Sub-task Resolved Charles Lamb
         
        21.
        Unify xattr name and value limits into a single limit Sub-task Resolved Andrew Wang
         
        22.
        Support extended attributes via WebHDFS Sub-task Resolved Yi Liu
         
        23.
        setXAttr should require the user to be the owner of the file or directory Sub-task Resolved Charles Lamb
         
        24.
        Should have different size limits for different XAttr namespaces (user, trusted, security and system). Sub-task Resolved Yi Liu
         
        25.
        xattr names erroneously handled as case-insensitive. Sub-task Resolved Charles Lamb
         
        26.
        xattr modification operations are based on state of latest snapshot instead of current version of inode. Sub-task Resolved Andrew Wang
         
        27.
        DFSClient unwraps AclException in xattr methods, but those methods cannot throw AclException. Sub-task Resolved Andrew Wang
         
        28.
        Interface audience and stability annotations missing from several new classes related to xattrs. Sub-task Resolved Andrew Wang
         

          Activity

          dhruba borthakur created issue -
          dhruba borthakur made changes -
          Field Original Value New Value
          Assignee dhruba borthakur [ dhruba ]
          dhruba borthakur made changes -
          Link This issue is related to HADOOP-7269 [ HADOOP-7269 ]
          Andrew Purtell made changes -
          Link This issue relates to HDFS-4672 [ HDFS-4672 ]
          Chris Nauroth made changes -
          Link This issue is required by HDFS-4685 [ HDFS-4685 ]
          Chris Nauroth made changes -
          Link This issue is required by HDFS-4685 [ HDFS-4685 ]
          Yi Liu made changes -
          Attachment xattrs.patch [ 12640385 ]
          Andrew Wang made changes -
          Assignee dhruba borthakur [ dhruba ] Yi Liu [ hitliuyi ]
          Andrew Wang made changes -
          Link This issue is required by HADOOP-10150 [ HADOOP-10150 ]
          Uma Maheswara Rao G made changes -
          Link This issue incorporates HADOOP-10514 [ HADOOP-10514 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-1.pdf [ 12640819 ]
          Yi Liu made changes -
          Attachment xattrs.1.patch [ 12640836 ]
          Yi Liu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Affects Version/s HDFS XAttrs [ 12326771 ]
          Uma Maheswara Rao G made changes -
          Comment [ [~cnauroth], thanks for the comments.
          {quote}
          As long as there is no way for the xattr APIs to "cross over" and touch things related to ACLs, then there is no risk of circumventing important ACL logic.
          {quote}

          Right.

          {quote}
          It sounded like you would take a previously unused bit on the inode (or a data structure referenced from the inode) and start toggling it on to indicate presence of xattrs. Do I understand correctly? If so, what specific data structure/field/bit do you have in mind?

          Earlier versions of the ACLs design doc described repurposing a previously unused bit in FsPermission as a flag to indicate presence of an ACL. We went through a lot of debate on this and finally decided not to do it. Instead, discovery of presence of an ACL is accomplished in the ls shell command via an extra RPC call. This is a trade-off of course, but the consensus was that this was preferable over implementing the ACL bit and introducing risk of data consistency bugs (i.e. if we store ACLs in the feature but forget to toggle on the ACL bit) or unexpected results (i.e. if some legacy bug accidentally turned on the thing that is now the ACL bit when persisting to fsimage). The only real need for a hasAcl kind of method was to support the ls display of appending '+' to the permission string when an ACL is present. I'm not aware of any similar requirement on ls for xattrs.
          {quota}

          “Whether inode has a XAttr is through a bit of int which is shared in inode” is only for phase of loading FSImage.
          The “hasXAttrs” is INodeSection.INodeFile#hasXAttrs, and only used when load FSImage. XAttrFeature is attached to an inode only if that inode has a XAttr. I will revise the doc to make it clear.


          {quota}
          I also can volunteer to help with code reviews on sub-tasks. Thank you for the replies!
          {quote}
          That would be great, very appreciate.

          The patches have been updated in sub-JIRAs, and Deign doc will be updated tomorrow. ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-2.pdf [ 12641724 ]
          Yi Liu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Yi Liu made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-2.pdf [ 12641724 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-2.pdf [ 12641852 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-3.pdf [ 12644341 ]
          Attachment Test-Plan-for-Extended-Attributes-1.pdf [ 12644342 ]
          Uma Maheswara Rao G made changes -
          Attachment ExtendedAttributes.html [ 12644787 ]
          Kihwal Lee made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2006-Merge-1.patch [ 12645057 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6410 [ HDFS-6410 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6412 [ HDFS-6412 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6413 [ HDFS-6413 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6414 [ HDFS-6414 ]
          Yi Liu made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2006-Merge-2.patch [ 12645209 ]
          Yi Liu made changes -
          Link This issue is related to HDFS-6426 [ HDFS-6426 ]
          Yi Liu made changes -
          Link This issue is related to HDFS-6430 [ HDFS-6430 ]
          Uma Maheswara Rao G made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6413 [ HDFS-6413 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6414 [ HDFS-6414 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6410 [ HDFS-6410 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6412 [ HDFS-6412 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HADOOP-10621 [ HADOOP-10621 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-6260 [ HDFS-6260 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-6375 [ HDFS-6375 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-6395 [ HDFS-6395 ]
          Yi Liu made changes -
          Link This issue relates to HDFS-6464 [ HDFS-6464 ]
          Yi Liu made changes -
          Link This issue relates to HDFS-6430 [ HDFS-6430 ]
          Yi Liu made changes -
          Link This issue is related to HDFS-6430 [ HDFS-6430 ]
          Yi Liu made changes -
          Link This issue relates to MAPREDUCE-5898 [ MAPREDUCE-5898 ]
          Yi Liu made changes -
          Link This issue relates to HDFS-6486 [ HDFS-6486 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2006-Branch-2-Merge.patch [ 12649598 ]
          Yi Liu made changes -
          Link This issue relates to MAPREDUCE-5920 [ MAPREDUCE-5920 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Incompatible change [ 10342 ] Reviewed [ 10343 ]
          Fix Version/s 2.5.0 [ 12326264 ]
          Resolution Fixed [ 1 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6556 [ HDFS-6556 ]
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]

            People

            • Assignee:
              Yi Liu
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development