Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14675

Increase Balancer Defaults Further

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: balancer & mover
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      Increase default bandwidth limit for rebalancing per DataNode (dfs.datanode.balance.bandwidthPerSec) from 10MB/s to 100MB/s.

      Increase default maximum threads of DataNode balancer (dfs.datanode.balance.max.concurrent.moves) from 50 to 100.
      Show
      Increase default bandwidth limit for rebalancing per DataNode (dfs.datanode.balance.bandwidthPerSec) from 10MB/s to 100MB/s. Increase default maximum threads of DataNode balancer (dfs.datanode.balance.max.concurrent.moves) from 50 to 100.

      Description

      HDFS-10297 increased the balancer defaults to 50 for dfs.datanode.balance.max.concurrent.moves and to 10MB/s for dfs.datanode.balance.bandwidthPerSec.

      We have found that these settings often have to be increased further as users find the balancer operates too slowly with 50 and 10MB/s. We often recommend moving concurrent moves to between 200 and 300 and setting the bandwidth to 100 or even 1000MB/s, and these settings seem to work well in practice.

      I would like to suggest we increase the balancer defaults further. I would suggest 100 for concurrent moves and 100MB/s for the bandwidth, but I would like to know what others think on this topic too.

        Attachments

        1. HDFS-14675.001.patch
          2 kB
          Stephen O'Donnell

          Activity

            People

            • Assignee:
              sodonnell Stephen O'Donnell
              Reporter:
              sodonnell Stephen O'Donnell
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: