Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12412

Change ErasureCodingWorker.stripedReadPool to cached thread pool

    Details

    • Target Version/s:
    • Release Note:
      Changed {{stripedReadPool}} to unbounded cachedThreadPool. User should combine {{dfs.datanode.ec.reconstruction.stripedblock.threads}} and {{dfs.namenode.replication.max-streams}} to tune recovery performance.

      Description

      In ErasureCodingWorker, it uses stripedReconstructionPool to schedule the EC recovery tasks, while uses stripedReadPool for the reader threads in each recovery task. We only need one of them to throttle the speed of recovery process, because each EC recovery task has a fix number of source readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the speed of EC recovery can be throttled by strippedReconstructionPool with xmitsInProgress.

      Moreover, keeping stripedReadPool makes customer difficult to understand and calculate the right balance between dfs.datanode.ec.reconstruction.stripedread.threads, dfs.datanode.ec.reconstruction.stripedblock.threads.size and maxReplicationStreams. For example, a small stripread.threads (comparing to which reconstruction.threads.size implies), will unnecessarily limit the speed of recovery, which leads to larger MTTR.

        Attachments

        1. HDFS-12412.00.patch
          3 kB
          Lei (Eddy) Xu
        2. HDFS-12412.01.patch
          5 kB
          Lei (Eddy) Xu

          Activity

            People

            • Assignee:
              eddyxu Lei (Eddy) Xu
              Reporter:
              eddyxu Lei (Eddy) Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: