Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3495

Update Balancer to support new NetworkTopology with NodeGroup

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: balancer & mover
    • Labels:
      None

      Description

      Since the Balancer is a Hadoop Tool, it was updated to be directly aware of four-layer hierarchy instead of creating an alternative Balancer implementation. To accommodate extensibility, a new protected method, doChooseNodesForCustomFaultDomain is now called from the existing chooseNodes method so that a subclass of the Balancer could customize the balancer algotirhm for other failure and locality topologies. An alternative option is to encapsulate the algorithm used for the four-layer hierarchy into a collaborating strategy class.
      The key changes introduced to support a four-layer hierarchy were to override the algorithm of choosing <source, target> pairs for balancing. Unit tests were created to test the new algorithm.
      The algorithm now makes sure to choose the target and source node on the same node group for balancing as the first priority. Then the overall balancing policy is: first doing balancing between nodes within the same nodegroup then the same rack and off rack at last. Also, we need to check no duplicated replicas live in the same node group after balancing.

        Attachments

        1. HADOOP-8473-Balancer-NodeGroup-aware.patch
          23 kB
          Junping Du
        2. HDFS-3495-branch-2.patch
          43 kB
          Junping Du
        3. HDFS-3495-v2.patch
          41 kB
          Junping Du
        4. HDFS-3495-v3.patch
          41 kB
          Junping Du
        5. HDFS-3495-v4.patch
          43 kB
          Junping Du
        6. HDFS-3495-v5.patch
          43 kB
          Junping Du

          Issue Links

            Activity

              People

              • Assignee:
                junping_du Junping Du
                Reporter:
                junping_du Junping Du
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: