Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12072

Provide fairness between EC and non-EC recovery tasks.

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      In DatanodeManager#handleHeartbeat, it takes up to maxTransfer reconstruction tasks for non-EC, then if the request can not be full filled, it takes more tasks from EC reconstruction tasks.

          List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
              maxTransfers);
          if (pendingList != null) {
            cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
                pendingList));
            maxTransfers -= pendingList.size();
          }
          // check pending erasure coding tasks
          List<BlockECReconstructionInfo> pendingECList = nodeinfo
              .getErasureCodeCommand(maxTransfers);
          if (pendingECList != null) {
            cmds.add(new BlockECReconstructionCommand(
                DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList));
          }
      

      So on a large cluster, if there are large number of constantly non-EC reconstruction tasks, EC reconstruction tasks do not have a chance to run.

      Attachments

        1. HDFS-12072.00.patch
          8 kB
          Lei (Eddy) Xu
        2. HDFS-12072.01.patch
          9 kB
          Lei (Eddy) Xu

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            eddyxu Lei (Eddy) Xu
            eddyxu Lei (Eddy) Xu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment