Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4992

Make balancer's thread count configurable

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: balancer & mover
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      By default, balancer has 1000 threads which moves blocks around (mover threads) and 200 threads which decide which block to move (dispatcher threads).

      On large clusters, 1000 threads creates significant load on NN, which slows down other HDFS activity. For example, on our cluster, 'hdfs dfs -ls /' command took about 1 minute when balancer is active. When no balancing in progress, the same command finishes in second or two.

      This patch makes amount of threads configurable by two new options 'dfs.balancer.moverThreads' and 'dfs.balancer.dispatcherThreads'.

        Attachments

        1. HDFS-4992-v1.patch
          2 kB
          Max Lapan
        2. HDFS-4992-v2.patch
          3 kB
          Max Lapan

          Activity

            People

            • Assignee:
              shmuma Max Lapan
              Reporter:
              shmuma Max Lapan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: