Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14563

Enhance interface about recommissioning/decommissioning

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: hdfs-client, namenode
    • Labels:

      Description

      In current implementation, if we need to decommissioning or recommissioning one datanode, the only way is add the datanode to include or exclude file under namenode configuration path then execute command `bin/hadoop dfsadmin -refreshNodes` and trigger namenode to reload include/exclude and start to recommissioning or decommissioning datanode.
      The shortcomings of this approach is that:
      a. namenode reload include/exclude configuration file from devices, if I/O load is high, handler may be blocked.
      b. namenode has to process every datnodes in include and exclude configurations, if there are many datanodes (very common for large cluster) pending to process, namenode will be hung for hundred seconds to wait recommision/decommision finish at the worst since holding write lock.
      I think we should expose one lightweight interface to support recommissioning or decommissioning single datanode, thus we can operate datanode using dfsadmin more smooth.

        Attachments

        1. mt_mode-2.txt
          22 kB
          Kihwal Lee
        2. HDFS-14563.002.patch
          39 kB
          Xiaoqiao He
        3. HDFS-14563.001.patch
          25 kB
          Xiaoqiao He

          Activity

            People

            • Assignee:
              hexiaoqiao Xiaoqiao He
              Reporter:
              hexiaoqiao Xiaoqiao He
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: