Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2006

ability to support storing extended attributes per file

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: HDFS XAttrs (HDFS-2006)
    • Fix Version/s: 2.5.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be nice if HDFS provides a feature to store extended attributes for files, similar to the one described here: http://en.wikipedia.org/wiki/Extended_file_attributes.
      The challenge is that it has to be done in such a way that a site not using this feature does not waste precious memory resources in the namenode.

      1. ExtendedAttributes.html
        22 kB
        Uma Maheswara Rao G
      2. HDFS-2006-Branch-2-Merge.patch
        278 kB
        Uma Maheswara Rao G
      3. HDFS-2006-Merge-1.patch
        258 kB
        Uma Maheswara Rao G
      4. HDFS-2006-Merge-2.patch
        266 kB
        Uma Maheswara Rao G
      5. HDFS-XAttrs-Design-1.pdf
        402 kB
        Yi Liu
      6. HDFS-XAttrs-Design-2.pdf
        424 kB
        Yi Liu
      7. HDFS-XAttrs-Design-3.pdf
        434 kB
        Yi Liu
      8. Test-Plan-for-Extended-Attributes-1.pdf
        305 kB
        Yi Liu
      9. xattrs.1.patch
        138 kB
        Yi Liu
      10. xattrs.patch
        129 kB
        Yi Liu

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          dhruba borthakur created issue -
          dhruba borthakur made changes -
          Field Original Value New Value
          Assignee dhruba borthakur [ dhruba ]
          dhruba borthakur made changes -
          Link This issue is related to HADOOP-7269 [ HADOOP-7269 ]
          Andrew Purtell made changes -
          Link This issue relates to HDFS-4672 [ HDFS-4672 ]
          Chris Nauroth made changes -
          Link This issue is required by HDFS-4685 [ HDFS-4685 ]
          Chris Nauroth made changes -
          Link This issue is required by HDFS-4685 [ HDFS-4685 ]
          Yi Liu made changes -
          Attachment xattrs.patch [ 12640385 ]
          Andrew Wang made changes -
          Assignee dhruba borthakur [ dhruba ] Yi Liu [ hitliuyi ]
          Andrew Wang made changes -
          Link This issue is required by HADOOP-10150 [ HADOOP-10150 ]
          Uma Maheswara Rao G made changes -
          Link This issue incorporates HADOOP-10514 [ HADOOP-10514 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-1.pdf [ 12640819 ]
          Yi Liu made changes -
          Attachment xattrs.1.patch [ 12640836 ]
          Yi Liu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Affects Version/s HDFS XAttrs [ 12326771 ]
          Uma Maheswara Rao G made changes -
          Comment [ [~cnauroth], thanks for the comments.
          {quote}
          As long as there is no way for the xattr APIs to "cross over" and touch things related to ACLs, then there is no risk of circumventing important ACL logic.
          {quote}

          Right.

          {quote}
          It sounded like you would take a previously unused bit on the inode (or a data structure referenced from the inode) and start toggling it on to indicate presence of xattrs. Do I understand correctly? If so, what specific data structure/field/bit do you have in mind?

          Earlier versions of the ACLs design doc described repurposing a previously unused bit in FsPermission as a flag to indicate presence of an ACL. We went through a lot of debate on this and finally decided not to do it. Instead, discovery of presence of an ACL is accomplished in the ls shell command via an extra RPC call. This is a trade-off of course, but the consensus was that this was preferable over implementing the ACL bit and introducing risk of data consistency bugs (i.e. if we store ACLs in the feature but forget to toggle on the ACL bit) or unexpected results (i.e. if some legacy bug accidentally turned on the thing that is now the ACL bit when persisting to fsimage). The only real need for a hasAcl kind of method was to support the ls display of appending '+' to the permission string when an ACL is present. I'm not aware of any similar requirement on ls for xattrs.
          {quota}

          “Whether inode has a XAttr is through a bit of int which is shared in inode” is only for phase of loading FSImage.
          The “hasXAttrs” is INodeSection.INodeFile#hasXAttrs, and only used when load FSImage. XAttrFeature is attached to an inode only if that inode has a XAttr. I will revise the doc to make it clear.


          {quota}
          I also can volunteer to help with code reviews on sub-tasks. Thank you for the replies!
          {quote}
          That would be great, very appreciate.

          The patches have been updated in sub-JIRAs, and Deign doc will be updated tomorrow. ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-2.pdf [ 12641724 ]
          Yi Liu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Yi Liu made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-2.pdf [ 12641724 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-2.pdf [ 12641852 ]
          Yi Liu made changes -
          Attachment HDFS-XAttrs-Design-3.pdf [ 12644341 ]
          Attachment Test-Plan-for-Extended-Attributes-1.pdf [ 12644342 ]
          Uma Maheswara Rao G made changes -
          Attachment ExtendedAttributes.html [ 12644787 ]
          Kihwal Lee made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2006-Merge-1.patch [ 12645057 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6410 [ HDFS-6410 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6412 [ HDFS-6412 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6413 [ HDFS-6413 ]
          Chris Nauroth made changes -
          Link This issue is related to HDFS-6414 [ HDFS-6414 ]
          Yi Liu made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2006-Merge-2.patch [ 12645209 ]
          Yi Liu made changes -
          Link This issue is related to HDFS-6426 [ HDFS-6426 ]
          Yi Liu made changes -
          Link This issue is related to HDFS-6430 [ HDFS-6430 ]
          Uma Maheswara Rao G made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6413 [ HDFS-6413 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6414 [ HDFS-6414 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6410 [ HDFS-6410 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6412 [ HDFS-6412 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HADOOP-10621 [ HADOOP-10621 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-6260 [ HDFS-6260 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-6375 [ HDFS-6375 ]
          Uma Maheswara Rao G made changes -
          Link This issue relates to HDFS-6395 [ HDFS-6395 ]
          Yi Liu made changes -
          Link This issue relates to HDFS-6464 [ HDFS-6464 ]
          Yi Liu made changes -
          Link This issue relates to HDFS-6430 [ HDFS-6430 ]
          Yi Liu made changes -
          Link This issue is related to HDFS-6430 [ HDFS-6430 ]
          Yi Liu made changes -
          Link This issue relates to MAPREDUCE-5898 [ MAPREDUCE-5898 ]
          Yi Liu made changes -
          Link This issue relates to HDFS-6486 [ HDFS-6486 ]
          Uma Maheswara Rao G made changes -
          Attachment HDFS-2006-Branch-2-Merge.patch [ 12649598 ]
          Yi Liu made changes -
          Link This issue relates to MAPREDUCE-5920 [ MAPREDUCE-5920 ]
          Uma Maheswara Rao G made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Incompatible change [ 10342 ] Reviewed [ 10343 ]
          Fix Version/s 2.5.0 [ 12326264 ]
          Resolution Fixed [ 1 ]
          Uma Maheswara Rao G made changes -
          Link This issue is related to HDFS-6556 [ HDFS-6556 ]
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]

            People

            • Assignee:
              Yi Liu
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development