Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15636

NameNode computes load by group when choosing datanodes.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd datanodes. The HOT files are stored with ALL_SSD and cold ones are stored with HOT. There is a big chance the NameNode couldn't choose enough nodes for writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A temporary solution isĀ  to increase the 'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the unbalance of load of all the datanodes.
      We should let the NameNode compute load by group. The ssd nodes and hdd nodes are computed separately and each group has its own average load. When the NameNode chooses a hdd node it only compares the node's load with
      the average load of the hdd group.

      Attachments

        Issue Links

          Activity

            People

              LiJinglun Jinglun
              LiJinglun Jinglun
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: