Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11419 BlockPlacementPolicyDefault is choosing datanode in an inefficient way
  3. HDFS-11514

DFSTopologyNodeImpl#chooseRandom optimizations

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 3.0.0-alpha4
    • namenode
    • None
    • Reviewed

    Description

      Based on the offline discussion, one potential improvement to the chooseRandomWithStorageType added in HDFS-11482 is that, currently given a node, the method iterates all its children to sum up the number of candidate datanodes. Since datanode status change is much less frequent than block placement request. It is more efficient to get rid of this iteration check, by probably maintaining another disk type counter map. This JIRA tracks (but not limited) this optimization.

      Attachments

        1. HDFS-11514.003.patch
          11 kB
          Chen Liang
        2. HDFS-11514.002.patch
          7 kB
          Chen Liang
        3. HDFS-11514.001.patch
          5 kB
          Chen Liang

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HDFS-11482 Add storage type demand to into DFSNetworkTopology#chooseRandom

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-12334 [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              vagarychen Chen Liang
              vagarychen Chen Liang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: