Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4959

Decommission data nodes, There is no response

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.5-alpha
    • Fix Version/s: None
    • Component/s: ha, hdfs-client, namenode
    • Labels:
      None

      Description

      There is "dfs.hosts.exclude" configured before NN start. Active/Standby works well.

      1)Add two datanodes IPAddr to the exclude file on the both Active and Standby NN.
      2)run: hdfs dfsadmin -refreshNodes on the Active NN, but there isn't any logs in the ActiveNN log. but decommission logs showed in the StandbyNN log.

      There is no decommission datanodes showed on the Active NN webUI. but there does have decommission datanodes showed on the Standby NN webUI.

      But the decommission process is very very slow, it indicates it cannot finish forever.

      I do think this is a bug.

        Activity

        Hide
        Zesheng Wu added a comment -

        A simple fix may look like this, but this is not a good fix:

          public void refreshNodes() throws IOException {
            namesystem.refreshNodes();
            namesystem.checkOperation(OperationCategory.WRITE);
          }
        
        Show
        Zesheng Wu added a comment - A simple fix may look like this, but this is not a good fix: public void refreshNodes() throws IOException { namesystem.refreshNodes(); namesystem.checkOperation(OperationCategory.WRITE); }
        Hide
        Zesheng Wu added a comment -

        Hi Fengdong, we have already encountered the same problem, and we think the root cause is as following I described:
        1. In the current implementation of dfs client, it uses a HashMap to store the NN addresses. The keys are host0 and host1, and values are the corresponding NN addresses. In Java HashMap, when traverse the HashMap, host1 is always ahead of host0.
        2. The dfs client traverses the HashMap to send requests to NN, at first it will request the host1 NN, if the request on host1 NN failed or the NN on host1 is standby, then dfs client will do failover and to request the host0 NN.
        3. In the current implementation of NN, nearly all the refresh work such as refreshNodes/refreshTopology/refreshUserToGroupsMappings donot check the state of NN. As a result, if the request is succeeded on the host1 NN, it will not try the host0 NN. But in our clusters, host0 NN is active and host1 NN is standby most of the time.

        We have already verified this in our cluster. And we think a complete implementation should send requests such as refreshNodes/refreshTopology/refreshUserToGroupsMappings to both the active and standby NN.

        Show
        Zesheng Wu added a comment - Hi Fengdong, we have already encountered the same problem, and we think the root cause is as following I described: 1. In the current implementation of dfs client, it uses a HashMap to store the NN addresses. The keys are host0 and host1, and values are the corresponding NN addresses. In Java HashMap, when traverse the HashMap, host1 is always ahead of host0. 2. The dfs client traverses the HashMap to send requests to NN, at first it will request the host1 NN, if the request on host1 NN failed or the NN on host1 is standby, then dfs client will do failover and to request the host0 NN. 3. In the current implementation of NN, nearly all the refresh work such as refreshNodes/refreshTopology/refreshUserToGroupsMappings donot check the state of NN. As a result, if the request is succeeded on the host1 NN, it will not try the host0 NN. But in our clusters, host0 NN is active and host1 NN is standby most of the time. We have already verified this in our cluster. And we think a complete implementation should send requests such as refreshNodes/refreshTopology/refreshUserToGroupsMappings to both the active and standby NN.
        Hide
        Fengdong Yu added a comment -

        Update:

        if add data_node_ip:50010 in the exclude file, then decommission works, but there are no decommissioned data nodes showed on the StandbyNN's web UI.

        Show
        Fengdong Yu added a comment - Update: if add data_node_ip:50010 in the exclude file, then decommission works, but there are no decommissioned data nodes showed on the StandbyNN's web UI.
        Hide
        Fengdong Yu added a comment -

        If stop standby NN, then run "hdfs dfsadmin -refreshNodes" on the Active NN, then decommission process is normal.

        Show
        Fengdong Yu added a comment - If stop standby NN, then run "hdfs dfsadmin -refreshNodes" on the Active NN, then decommission process is normal.

          People

          • Assignee:
            Unassigned
            Reporter:
            Fengdong Yu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development