Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12072

Provide fairness between EC and non-EC recovery tasks.

    Details

      Description

      In DatanodeManager#handleHeartbeat, it takes up to maxTransfer reconstruction tasks for non-EC, then if the request can not be full filled, it takes more tasks from EC reconstruction tasks.

          List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
              maxTransfers);
          if (pendingList != null) {
            cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
                pendingList));
            maxTransfers -= pendingList.size();
          }
          // check pending erasure coding tasks
          List<BlockECReconstructionInfo> pendingECList = nodeinfo
              .getErasureCodeCommand(maxTransfers);
          if (pendingECList != null) {
            cmds.add(new BlockECReconstructionCommand(
                DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList));
          }
      

      So on a large cluster, if there are large number of constantly non-EC reconstruction tasks, EC reconstruction tasks do not have a chance to run.

        Attachments

        1. HDFS-12072.00.patch
          8 kB
          Lei (Eddy) Xu
        2. HDFS-12072.01.patch
          9 kB
          Lei (Eddy) Xu

          Activity

            People

            • Assignee:
              eddyxu Lei (Eddy) Xu
              Reporter:
              eddyxu Lei (Eddy) Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: