Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11565

Use compact identifiers for built-in ECPolicies in HdfsFileStatus

    XMLWordPrintableJSON

Details

    • Incompatible change
    • Some of the existing fields in ErasureCodingPolicyProto have changed from required to optional. For system EC policies, these fields are populated from hardcoded values.

    Description

      Discussed briefly on HDFS-7337 with Kai Zheng. Quoting our convo:

      From looking at the protos, one other question I had is about the overhead of these protos when using the hardcoded policies. There are a bunch of strings and ints, which can be kind of heavy since they're added to each HdfsFileStatus. Should we make the built-in ones identified by purely an ID, with these fully specified protos used for the pluggable policies?

      Sounds like this could be considered separately because, either built-in policies or plugged-in polices, the full meta info is maintained either by the codes or in the fsimage persisted, so identifying them by purely an ID should works fine. If agree, we could refactor the codes you mentioned above separately.

      Attachments

        1. HDFS-11565.001.patch
          7 kB
          Andrew Wang
        2. HDFS-11565.002.patch
          7 kB
          Andrew Wang
        3. HDFS-11565.003.patch
          9 kB
          Andrew Wang

        Issue Links

          Activity

            People

              andrew.wang Andrew Wang
              andrew.wang Andrew Wang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: