Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8900

Compact XAttrs to optimize memory footprint.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The config key "dfs.namenode.fs-limits.max-xattr-size" can no longer be set to a value of 0 (previously used to indicate unlimited) or a value greater than 32KB. This is a constraint on xattr size similar to many local filesystems.

      Description

      private final ImmutableList<XAttr> xAttrs;
      

      Currently we use above in XAttrFeature, it's not efficient from memory point of view, since ImmutableList and XAttr have object memory overhead, and each object has memory alignment.

      We can use a byte[] in XAttrFeature and do some compact in XAttr.

        Attachments

        1. HDFS-8900.001.patch
          32 kB
          Yi Liu
        2. HDFS-8900.002.patch
          37 kB
          Yi Liu
        3. HDFS-8900.003.patch
          39 kB
          Yi Liu
        4. HDFS-8900.004.patch
          41 kB
          Yi Liu
        5. HDFS-8900.005.patch
          41 kB
          Yi Liu

          Issue Links

            Activity

              People

              • Assignee:
                hitliuyi Yi Liu
                Reporter:
                hitliuyi Yi Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: