[HDFS-8461] Erasure coding: fix priority level of UnderReplicatedBlocks for striped block

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: HDFS-7285
• Hadoop Flags: Reviewed
• Assignee: Walter Su
• Reporter: Walter Su
• Created: 22/May/15 06:40
• Updated: 21/Dec/15 18:35
• Resolved: 06/Jul/15 23:40

Description

Issues 1: correctly mark corrupted blocks.
Issues 2: distinguish highest risk priority and normal risk priority.

UnderReplicatedBlocks.java

  private int getPriority(int curReplicas,
  ...
    } else if (curReplicas == 1) {
      //only on replica -risk of loss
      // highest priority
      return QUEUE_HIGHEST_PRIORITY;
  ...

For stripe blocks, we should return QUEUE_HIGHEST_PRIORITY when curReplicas == 6( Suppose 6+3 schema).

That's important. Because

BlockManager.java

DatanodeDescriptor[] chooseSourceDatanodes(BlockInfo block,
  ...
     if(priority != UnderReplicatedBlocks.QUEUE_HIGHEST_PRIORITY 
          && !node.isDecommissionInProgress() 
          && node.getNumberOfBlocksToBeReplicated() >= maxReplicationStreams)
      {
        continue; // already reached replication limit
      }
  ...

It may return not enough source DNs ( maybe 5), and failed to recover.
A busy node should not be skiped if a block has highest risk/priority. The issue is the striped block doesn't have priority.

Issue Links

• duplicates: HDFS-8010 Erasure coding: extend UnderReplicatedBlocks to accurately handle striped blocks
• is related to: HDFS-7354 Optimized handling of parity blocks in block management
• requires: HDFS-8567 Erasure Coding: SafeMode handles file smaller than a full stripe