Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8468 Umbrella of enhancements to support different failure and locality topologies
  3. HADOOP-8470

Implementation of 4-layer subclass of NetworkTopology (NetworkTopologyWithNodeGroup)

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: None
    • Labels:
      None

      Description

      To support the four-layer hierarchical topology shown in attached figure as a subclass of NetworkTopology, NetworkTopologyWithNodeGroup was developed along with unit tests. NetworkTopologyWithNodeGroup overriding the methods add, remove, and pseudoSortByDistance were the most relevant to support the four-layer topology. The method seudoSortByDistance selects the nodes to use for reading data and sorts the nodes in sequence of node-local, nodegroup-local, rack- local, rack–off. Another slightly change to seudoSortByDistance is to support cases of separation data node and node manager: if the reader cannot be found in NetworkTopology tree (formed by data nodes only), then it will try to sort according to reader's sibling node in the tree.
      The distance calculation changes the weights from 0 (local), 2 (rack- local), 4 (rack-off) to: 0 (local), 2 (nodegroup-local), 4 (rack-local), 6 (rack-off).
      The additional node group layer should be specified in the topology script or table mapping, e.g. input 10.1.1.1, output: /rack1/nodegroup1
      A subclass on InnerNode, InnerNodeWithNodeGroup, was also needed to support NetworkTopologyWithNodeGroup.

        Attachments

        1. HADOOP-8470-NetworkTopology-new-impl-v4.patch
          21 kB
          Junping Du
        2. HADOOP-8470-NetworkTopology-new-impl-v3.patch
          21 kB
          Junping Du
        3. HADOOP-8470-NetworkTopology-new-impl-v2.patch
          21 kB
          Junping Du
        4. HADOOP-8470-NetworkTopology-new-impl.patch
          22 kB
          Junping Du
        5. HADOOP-8470-branch-2.patch
          21 kB
          Junping Du

        Issue Links

          Activity

            People

            • Assignee:
              junping_du Junping Du
              Reporter:
              junping_du Junping Du

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment