[HADOOP-4061] Large number of decommission freezes the Namenode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.2
• Fix Version/s: 0.18.3, 0.19.1
• Hadoop Flags: Incompatible change, Reviewed

Release Note

Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval.

• Assignee: Tsz-wo Sze
• Reporter: Koji Noguchi
• Created: 03/Sep/08 18:03
• Updated: 08/Jul/09 16:43
• Resolved: 25/Nov/08 18:06

Description

On 1900 nodes cluster, we tried decommissioning 400 nodes with 30k blocks each. Other 1500 nodes were almost empty.

When decommission started, namenode's queue overflowed every 6 minutes.

Looking at the cpu usage, it showed that every 5 minutes org.apache.hadoop.dfs.FSNamesystem$DecommissionedMonitor thread was taking 100% of the CPU for 1 minute causing the queue to overflow.

  public synchronized void decommissionedDatanodeCheck() {
    for (Iterator<DatanodeDescriptor> it = datanodeMap.values().iterator();
         it.hasNext();) {
      DatanodeDescriptor node = it.next();
      checkDecommissionStateInternal(node);
    }
  }

Issue Links

• is related to: HDFS-283 Improve datanode decommission monitoring performance