Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10206

Datanodes not sorted properly by distance when the reader isn't a datanode


    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      If the DFSClient machine is not a datanode, but it shares its rack with some datanodes of the HDFS block requested, DatanodeManager#sortLocatedBlocks might not put the local-rack datanodes at the beginning of the sorted list. That is because the function didn't call networktopology.add(client); to properly set the node's parent node; something required by networktopology.sortByDistance to compute distance between two nodes in the same topology tree.

      Another issue with networktopology.sortByDistance is it only distinguishes local rack from remote rack, but it doesn't support general distance calculation to tell how remote the rack is.

        protected int getWeight(Node reader, Node node) {
          // 0 is local, 1 is same rack, 2 is off rack
          // Start off by initializing to off rack
          int weight = 2;
          if (reader != null) {
            if (reader.equals(node)) {
              weight = 0;
            } else if (isOnSameRack(reader, node)) {
              weight = 1;
          return weight;

      HDFS-10203 has suggested moving the sorting from namenode to DFSClient to address another issue. Regardless of where we do the sorting, we still need fix the issues outline here.

      Note that BlockPlacementPolicyDefault shares the same NetworkTopology object used by DatanodeManager and requires Nodes stored in the topology to be DatanodeDescriptor for block placement. So we need to make sure we don't pollute the NetworkTopology if we plan to fix it on the server side.


        1. HDFS-10206.000.patch
          6 kB
          Nanda kumar
        2. HDFS-10206.001.patch
          11 kB
          Nanda kumar
        3. HDFS-10206.002.patch
          11 kB
          Nanda kumar
        4. HDFS-10206.003.patch
          11 kB
          Nanda kumar
        5. HDFS-10206-branch-2.8.003.patch
          11 kB
          Nanda kumar



            • Assignee:
              nandakumar131 Nanda kumar
              mingma Ming Ma
            • Votes:
              0 Vote for this issue
              10 Start watching this issue


              • Created: