[HDFS-12072] Provide fairness between EC and non-EC recovery tasks.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha3
• Fix Version/s: 3.0.0-beta1
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-nice-to-have
• Target Version/s: 3.0.0-beta1
• Assignee: Lei (Eddy) Xu
• Reporter: Lei (Eddy) Xu
• Created: 29/Jun/17 23:47
• Updated: 17/Aug/17 22:55
• Resolved: 17/Aug/17 22:26

Description

In DatanodeManager#handleHeartbeat, it takes up to maxTransfer reconstruction tasks for non-EC, then if the request can not be full filled, it takes more tasks from EC reconstruction tasks.

    List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
        maxTransfers);
    if (pendingList != null) {
      cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
          pendingList));
      maxTransfers -= pendingList.size();
    }
    // check pending erasure coding tasks
    List<BlockECReconstructionInfo> pendingECList = nodeinfo
        .getErasureCodeCommand(maxTransfers);
    if (pendingECList != null) {
      cmds.add(new BlockECReconstructionCommand(
          DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList));
    }

So on a large cluster, if there are large number of constantly non-EC reconstruction tasks, EC reconstruction tasks do not have a chance to run.