Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12044

Mismatch between BlockManager#maxReplicationStreams and ErasureCodingWorker.stripedReconstructionPool pool size causes slow and bursty recovery

    Details

      Description

      ErasureCodingWorker#stripedReconstructionPool is with corePoolSize=2 and maxPoolSize=8 as default. And it rejects more tasks if the queue is full.

      When BlockManager#maxReplicationStream is larger than ErasureCodingWorker#stripedReconstructionPool#corePoolSize/maxPoolSize, for example, maxReplicationStream=20 and corePoolSize=2 , maxPoolSize=8. Meanwhile, NN sends up to maxTransfer reconstruction tasks to DN for each heartbeat, and it is calculated in FSNamesystem:

      final int maxTransfer = blockManager.getMaxReplicationStreams() - xmitsInProgress;
      

      However, at any giving time, {ErasureCodingWorker#stripedReconstructionPool takes 2 xmitInProcess. So for each heartbeat in 3s, NN will send about 20-2 = 18 reconstruction tasks to the DN, and DN throw away most of them if there were 8 tasks in the queue already. So NN needs to take longer to re-consider these blocks were under-replicated to schedule new tasks.

        Attachments

        1. HDFS-12044.00.patch
          1 kB
          Lei (Eddy) Xu
        2. HDFS-12044.01.patch
          5 kB
          Lei (Eddy) Xu
        3. HDFS-12044.02.patch
          8 kB
          Lei (Eddy) Xu
        4. HDFS-12044.03.patch
          14 kB
          Lei (Eddy) Xu
        5. HDFS-12044.04.patch
          17 kB
          Lei (Eddy) Xu
        6. HDFS-12044.05.patch
          17 kB
          Lei (Eddy) Xu

          Issue Links

            Activity

              People

              • Assignee:
                eddyxu Lei (Eddy) Xu
                Reporter:
                eddyxu Lei (Eddy) Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: