Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-8580

Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HDFS-7285
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought: Add a cellSize field in BlockInfoStriped as a workaround, and deal with memory usage in follow-on.(HDFS-8059)

      discussion in HDFS-8494:
      from Walter Su:

      I think BlockInfoStriped needs to keep cellSize.

      from [~vinayrpet]:

      I too was thinking the same when the FSImageLoader problem has came up. This will increase the memory usage by ~4bytes for each block though.

      from Jing Zhao

      Also, we should consider adding a chunk size field to StripedBlockProto and removing the cell size field from HdfsFileStatus. In this way we can access the chunk size information in the storage layer.

      ==
      update:
      from Jing Zhao

      For fsimage part, since HDFS-8585 just removes StripedBlockProto, I guess what we can do here is to either 1) add the cellSize information into StripedBlocksFeature in fsimage.proto, or 2) bring StripedBlockProto back and put block info and cell size there.

        Attachments

        1. HDFS-8580-HDFS-7285.04.patch
          29 kB
          Walter Su
        2. HDFS-8580-HDFS-7285.03.patch
          29 kB
          Walter Su
        3. HDFS-8580-HDFS-7285.02.patch
          29 kB
          Walter Su
        4. HDFS-8580-HDFS-7285.01.patch
          28 kB
          Walter Su
        5. HDFS-8580.00.patch
          30 kB
          Walter Su

          Issue Links

            Activity

              People

              • Assignee:
                walter.k.su Walter Su
                Reporter:
                walter.k.su Walter Su
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: