Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0-alpha3
-
Changed {{stripedReadPool}} to unbounded cachedThreadPool. User should combine {{dfs.datanode.ec.reconstruction.stripedblock.threads}} and {{dfs.namenode.replication.max-streams}} to tune recovery performance.
Description
In ErasureCodingWorker, it uses stripedReconstructionPool to schedule the EC recovery tasks, while uses stripedReadPool for the reader threads in each recovery task. We only need one of them to throttle the speed of recovery process, because each EC recovery task has a fix number of source readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the speed of EC recovery can be throttled by strippedReconstructionPool with xmitsInProgress.
Moreover, keeping stripedReadPool makes customer difficult to understand and calculate the right balance between dfs.datanode.ec.reconstruction.stripedread.threads, dfs.datanode.ec.reconstruction.stripedblock.threads.size and maxReplicationStreams. For example, a small stripread.threads (comparing to which reconstruction.threads.size implies), will unnecessarily limit the speed of recovery, which leads to larger MTTR.