[HDFS-11382] Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha4
• Component/s: hdfs
• Target Version/s: 3.0.0-alpha4
• Hadoop Flags: Incompatible change

Release Note

The FSImage on-disk format for INodeFile is changed to additionally include a field for Erasure Coded files. This optional field 'erasureCodingPolicyID' which is unit32 type is available for all Erasure Coded files and represents the Erasure Coding Policy ID. Previously, the 'replication' field in INodeFile disk format was overloaded to represent the same Erasure Coding Policy ID.

• Assignee: Manoj Govindassamy
• Reporter: Manoj Govindassamy
• Created: 31/Jan/17 18:58
• Updated: 28/Feb/17 14:00
• Resolved: 28/Feb/17 01:08

Description

For Erasure Coded files, replication field in INodeFile message is re-used to store the EC Policy ID.

FSDirWriteFileOp#addFile

  private static INodesInPath addFile(
      FSDirectory fsd, INodesInPath existing, byte[] localName,
      PermissionStatus permissions, short replication, long preferredBlockSize,
      String clientName, String clientMachine)
      throws IOException {
    .. .. ..
    try {
      ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
          getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
      if (ecPolicy != null) {
        replication = ecPolicy.getId();   <===
      }
      final BlockType blockType = ecPolicy != null?
          BlockType.STRIPED : BlockType.CONTIGUOUS;
      INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
          modTime, modTime, replication, preferredBlockSize, blockType);
      newNode.setLocalName(localName);
      newNode.toUnderConstruction(clientName, clientMachine);
      newiip = fsd.addINode(existing, newNode, permissions.getPermission());

With HDFS-11268 fix, FSImageFormatPBINode#Loader#loadInodeFile is rightly getting the EC ID from the replication field and then uses the right Policy to construct the blocks.
FSImageFormatPBINode#Loader#loadInodeFile

      ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
          ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :
          null;

The original intention was to re-use the replication field so the in-memory representation would be compact. But, this isn't necessary for the on-disk representation. replication is an optional field, and if we add another optional field for the EC policy, it won't be any extra space.

Also, we need to make sure to have the appropriate asserts in place to make sure both fields aren't set for the same INodeField.

Issue Links

• is related to: HDFS-10867 [PROVIDED Phase 2] Block Bit Field Allocation of Provided Storage
• is related to: HDFS-11465 Rename BlockType#CONTIGUOUS to BlockType#REPLICATED