Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-16613

EC: Improve performance of decommissioning dn with many ec blocks

    XMLWordPrintableJSON

Details

    Description

      In a hdfs cluster with a lot of EC blocks, decommission a dn is very slow. The reason is unlike replication blocks can be replicated from any dn which has the same block replication, the ec block have to be replicated from the decommissioning dn.

      The configurations dfs.namenode.replication.max-streams and dfs.namenode.replication.max-streams-hard-limit will limit the replication speed, but increase these configurations will create risk to the whole cluster's network. So it should add a new configuration to limit the decommissioning dn, distinguished from the cluster wide max-streams limit.

      Attachments

        1. image-2022-06-07-11-46-42-389.png
          51 kB
          caozhiqiang
        2. image-2022-06-07-17-42-16-075.png
          175 kB
          caozhiqiang
        3. image-2022-06-07-17-45-45-316.png
          195 kB
          caozhiqiang
        4. image-2022-06-07-17-51-04-876.png
          361 kB
          caozhiqiang
        5. image-2022-06-07-17-55-40-203.png
          181 kB
          caozhiqiang
        6. image-2022-06-08-11-38-29-664.png
          159 kB
          caozhiqiang
        7. image-2022-06-08-11-41-11-127.png
          137 kB
          caozhiqiang

        Issue Links

          Activity

            People

              caozhiqiang caozhiqiang
              caozhiqiang caozhiqiang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2.5h
                  2.5h