Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-8833

Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: HDFS-7285
    • Fix Version/s: HDFS-7285
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We have discussed storing EC schema with files instead of EC zones and recently revisited the discussion under HDFS-8059.

      As a recap, the zone concept has severe limitations including renaming and nested configuration. Those limitations are valid in encryption for security reasons and it doesn't make sense to carry them over in EC.

      This JIRA aims to store EC schema and cell size on INodeFile level. For simplicity, we should first implement it as an xattr and consider memory optimizations (such as moving it to file header) as a follow-on. We should also disable changing EC policy on a non-empty file / dir in the first phase.

        Attachments

        1. HDFS-8833-HDFS-7285.08.patch
          142 kB
          Zhe Zhang
        2. HDFS-8833-HDFS-7285.07.patch
          138 kB
          Zhe Zhang
        3. HDFS-8833-HDFS-7285.06.patch
          138 kB
          Zhe Zhang
        4. HDFS-8833-HDFS-7285.05.patch
          137 kB
          Zhe Zhang
        5. HDFS-8833-HDFS-7285.04.patch
          135 kB
          Zhe Zhang
        6. HDFS-8833-HDFS-7285.03.patch
          138 kB
          Zhe Zhang
        7. HDFS-8833-HDFS-7285.02.patch
          139 kB
          Zhe Zhang
        8. HDFS-8833-HDFS-7285-merge.01.patch
          123 kB
          Zhe Zhang
        9. HDFS-8833-HDFS-7285-merge.00.patch
          99 kB
          Zhe Zhang

          Issue Links

            Activity

              People

              • Assignee:
                zhz Zhe Zhang
                Reporter:
                zhz Zhe Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: