[HDFS-14350] dfs.datanode.ec.reconstruction.threads not take effect

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.2.0
• Component/s: datanode, ec
• Labels: pull-request-available
• Assignee: hunshenshi
• Reporter: hunshenshi
• Created: 09/Mar/19 05:28
• Updated: 24/Jul/21 06:08
• Resolved: 04/May/21 12:00

Description

In ErasureCodingWorker, stripedReconstructionPool is create by 

initializeStripedBlkReconstructionThreadPool(conf.getInt(
    DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_KEY,
    DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_DEFAULT));

private void initializeStripedBlkReconstructionThreadPool(int numThreads) {
  LOG.debug("Using striped block reconstruction; pool threads={}",
      numThreads);
  stripedReconstructionPool = DFSUtilClient.getThreadPoolExecutor(2,
      numThreads, 60, new LinkedBlockingQueue<>(),
      "StripedBlockReconstruction-", false);
  stripedReconstructionPool.allowCoreThreadTimeOut(true);
}

so stripedReconstructionPool is a ThreadPoolExecutor, and the queue is a LinkedBlockingQueue, then the active thread is awalys 2, the dfs.datanode.ec.reconstruction.threads not take effect.

 

Issue Links

• duplicates: HDFS-14367 EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads
• links to: GitHub Pull Request #582