Hadoop Common
  1. Hadoop Common
  2. HADOOP-4061

Large number of decommission freezes the Namenode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.17.2
    • Fix Version/s: 0.18.3, 0.19.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval.
      Show
      Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval.

      Description

      On 1900 nodes cluster, we tried decommissioning 400 nodes with 30k blocks each. Other 1500 nodes were almost empty.

      When decommission started, namenode's queue overflowed every 6 minutes.

      Looking at the cpu usage, it showed that every 5 minutes org.apache.hadoop.dfs.FSNamesystem$DecommissionedMonitor thread was taking 100% of the CPU for 1 minute causing the queue to overflow.

        public synchronized void decommissionedDatanodeCheck() {
          for (Iterator<DatanodeDescriptor> it = datanodeMap.values().iterator();
               it.hasNext();) {
            DatanodeDescriptor node = it.next();
            checkDecommissionStateInternal(node);
          }
        }
      
      1. 4061_20081124c_0.18.patch
        18 kB
        Tsz Wo Nicholas Sze
      2. 4061_20081124c.patch
        20 kB
        Tsz Wo Nicholas Sze
      3. HADOOP-4061.patch
        13 kB
        Raghu Angadi
      4. 4061_20081124b.patch
        19 kB
        Tsz Wo Nicholas Sze
      5. 4061_20081124.patch
        20 kB
        Tsz Wo Nicholas Sze
      6. 4061_20081123.patch
        17 kB
        Tsz Wo Nicholas Sze
      7. 4061_20081120b.patch
        13 kB
        Tsz Wo Nicholas Sze
      8. 4061_20081120.patch
        13 kB
        Tsz Wo Nicholas Sze
      9. 4061_20081119.patch
        12 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Koji Noguchi created issue -
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Attachment 4061_20081119.patch [ 12394286 ]
          Tsz Wo Nicholas Sze made changes -
          Release Note Added a new conf property dfs.namenode.decommission.nodes.per.interval so that NameNode checks decommission status of x nodes for every y seconds, where x is the value of dfs.namenode.decommission.nodes.per.interval and y is the value of dfs.namenode.decommission.interval.
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
          Hadoop Flags [Incompatible change]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081120.patch [ 12394374 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HADOOP-4701 [ HADOOP-4701 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081120b.patch [ 12394382 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081123.patch [ 12394497 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081124.patch [ 12394591 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081124b.patch [ 12394602 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Raghu Angadi made changes -
          Attachment HADOOP-4061.patch [ 12394603 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081124c.patch [ 12394613 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment 4061_20081124c_0.18.patch [ 12394672 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed, Incompatible change] [Incompatible change, Reviewed]
          Fix Version/s 0.18.3 [ 12313494 ]
          Fix Version/s 0.20.0 [ 12313438 ]
          Fix Version/s 0.19.1 [ 12313473 ]
          Nigel Daley made changes -
          Fix Version/s 0.20.0 [ 12313438 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s dfs [ 12310710 ]

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development