Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1773

Remove a datanode from cluster if include list is not empty and this datanode is removed from both include and exclude lists

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.1, 0.23.0
    • Fix Version/s: 0.20.204.0, 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Environment:

      branch-20-security and trunk.

    • Hadoop Flags:
      Reviewed
    • Tags:
      datanode, web console, decommission data node

      Description

      Our service engineering team who operates the clusters on a daily basis founds it is confusing that after a data node is decommissioned, there is no way to make the cluster forget about this data node and it always remains in the dead node list.

      1. HDFS-1867.patch
        5 kB
        Tanping Wang
      2. HDFS-1773-3.patch
        3 kB
        Tanping Wang
      3. HDFS-1773-2.patch
        3 kB
        Tanping Wang
      4. HDFS-1773.patch
        2 kB
        Tanping Wang

        Issue Links

          Activity

          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was released.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #673 (See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/673/ )
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed the patch for trunk. Thanks, Tanping!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed the patch for trunk. Thanks, Tanping!
          Hide
          Tanping Wang added a comment -

          The failed core test cases are irrelevant to this patch. Among them,

          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader

          are also seen failing with other just posted patches.

          org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
          org.apache.hadoop.hdfs.TestReadWhileWriting

          I run them manually and they are passing.

          Show
          Tanping Wang added a comment - The failed core test cases are irrelevant to this patch. Among them, org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader are also seen failing with other just posted patches. org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage org.apache.hadoop.hdfs.TestReadWhileWriting I run them manually and they are passing.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477991/HDFS-1867.patch
          against trunk revision 1098867.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.hdfs.server.namenode.TestBackupNode
          org.apache.hadoop.hdfs.TestDatanodeBlockScanner
          org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
          org.apache.hadoop.hdfs.TestFileConcurrentReader
          org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
          org.apache.hadoop.hdfs.TestReadWhileWriting

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/446//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/446//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/446//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477991/HDFS-1867.patch against trunk revision 1098867. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestDatanodeBlockScanner org.apache.hadoop.hdfs.TestDFSStorageStateRecovery org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage org.apache.hadoop.hdfs.TestReadWhileWriting +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/446//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/446//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/446//console This message is automatically generated.
          Hide
          Tanping Wang added a comment -

          Run test-patch manually,

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no new tests are needed for this patch.
          [exec] Also please list what manual steps were performed to verify this patch.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          [exec]
          [exec] +1 system test framework. The patch passed system test framework compile.
          [exec]
          [exec]
          [exec]
          [exec]
          [exec] ======================================================================
          [exec] ======================================================================
          [exec] Finished build.
          [exec] ======================================================================
          [exec] ======================================================================

          No unit test added, but tested manually.

          Show
          Tanping Wang added a comment - Run test-patch manually, [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== No unit test added, but tested manually.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 the patch for trunk looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 the patch for trunk looks good.
          Hide
          Tanping Wang added a comment -

          Upload the patch generated for trunk(with Federation).

          "Hide/not display/forget" an already decommissioned data node who is neither in include nor exclude hosts lists from the list of live or dead nodes. This is being found useful for operators to make the cluster forget a decommissioned data node. The operation procedure is as following:
          Step I: Host must have been in the include hosts list and the include hosts list must not be empty.
          Step II: Host is decommissioned by remaining in the include hosts list and added into the exclude hosts list. Name node is updated with the new information by issuing dfsadmin -refreshNodes command.
          Step III: Host is removed from both include hosts and exclude hosts lists. Name node is updated with the new information by issuing dfsamin -refreshNodes command.

          Couple variations to consider:

          1. Step II variation ( all the other steps are the same), if a node is added into exclude list, but removed from include list, the node will appear as dead node.
          2. Step III variation (all the other steps are the same), if a node remains in
          the include list, but removed from exclude list, the node will rejoin as a
          live node.

          Show
          Tanping Wang added a comment - Upload the patch generated for trunk(with Federation). "Hide/not display/forget" an already decommissioned data node who is neither in include nor exclude hosts lists from the list of live or dead nodes. This is being found useful for operators to make the cluster forget a decommissioned data node. The operation procedure is as following: Step I: Host must have been in the include hosts list and the include hosts list must not be empty. Step II: Host is decommissioned by remaining in the include hosts list and added into the exclude hosts list. Name node is updated with the new information by issuing dfsadmin -refreshNodes command. Step III: Host is removed from both include hosts and exclude hosts lists. Name node is updated with the new information by issuing dfsamin -refreshNodes command. Couple variations to consider: 1. Step II variation ( all the other steps are the same), if a node is added into exclude list, but removed from include list, the node will appear as dead node. 2. Step III variation (all the other steps are the same), if a node remains in the include list, but removed from exclude list, the node will rejoin as a live node.
          Hide
          Tanping Wang added a comment -

          Port this jira to trunk. Mark HDFS-1867 as a duplication of this jira.

          Show
          Tanping Wang added a comment - Port this jira to trunk. Mark HDFS-1867 as a duplication of this jira.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this to 0.20-security. Thanks, Tanping!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this to 0.20-security. Thanks, Tanping!
          Hide
          Tanping Wang added a comment -

          I run test-patch with my patch
          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
          [exec] Please justify why no tests are needed for this patch.
          [exec]
          [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
          [exec]
          [exec]
          [exec]
          [exec]
          [exec] ======================================================================
          [exec] ======================================================================
          [exec] Finished build.
          [exec] ======================================================================
          [exec] ======================================================================
          [exec]
          [exec]

          So I went ahead run another test-patch with a fake patch whose content is empty. And I got the same result. I have also run $ant javadoc with and without my patch and I got total of 31 javadoc warnings with out without my patch.

          I also run ant test. With and without my patch, I have seen the follow tests failed on branch-20-security,

          [junit] Test org.apache.hadoop.util.TestQueueProcessingStatistics FAILED
          [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED

          These two tests are not related with my patch.

          Show
          Tanping Wang added a comment - I run test-patch with my patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] -1 javadoc. The javadoc tool appears to have generated 1 warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] So I went ahead run another test-patch with a fake patch whose content is empty. And I got the same result. I have also run $ant javadoc with and without my patch and I got total of 31 javadoc warnings with out without my patch. I also run ant test. With and without my patch, I have seen the follow tests failed on branch-20-security, [junit] Test org.apache.hadoop.util.TestQueueProcessingStatistics FAILED [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED These two tests are not related with my patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. Please run test-patch and the unit tests.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. Please run test-patch and the unit tests.
          Hide
          Tanping Wang added a comment -

          Thanks for the comments, Nicholas. I have made changes based on the comments and uploaded the new patch.

          Show
          Tanping Wang added a comment - Thanks for the comments, Nicholas. I have made changes based on the comments and uploaded the new patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In removeDecomNodeFromDeadList(..),

          • remove public.
          • hostsReader.getHosts() is independent of the nodes. If it is empty, removeDecomNodeFromDeadList(..) can return immediately. So we should check it before the loop.
          • fsn is not a parameter.
            +   * @param fsn, FSNamesystem
            
          Show
          Tsz Wo Nicholas Sze added a comment - In removeDecomNodeFromDeadList(..) , remove public. hostsReader.getHosts() is independent of the nodes. If it is empty, removeDecomNodeFromDeadList(..) can return immediately. So we should check it before the loop. fsn is not a parameter. + * @param fsn, FSNamesystem
          Hide
          Tanping Wang added a comment -

          Discussed with Nicholas about the first approach by removing datanode from datanodeMap directly. The concern here is that datanodeMap is a super set that stores the datanode -> block map. If we remove the datanode from the datanodeMap, we are not certain if any potential negative impact would happen to some other related data structures whose relationship with datanodeMap is not that obvious. Plus, it seems that a dedicated function, wipeDatanode(DatanodeID nodeID) is being used to remove a node from datanodeMap. Since this is only for 20
          release, ( decommission data states will be changed in next release of HDFS), we decide to only remove the decommission node from the list of dead nodes for displaying purpose only. That is to say, before displaying the list of dead node to either JSP or jmx, we check for decommissioned dead data node and remove them for displaying. We also add a check

          node.isDecommissioned())
          

          to make sure before removing the data node from dead list, it is already
          decommissioned.

          Show
          Tanping Wang added a comment - Discussed with Nicholas about the first approach by removing datanode from datanodeMap directly. The concern here is that datanodeMap is a super set that stores the datanode -> block map. If we remove the datanode from the datanodeMap, we are not certain if any potential negative impact would happen to some other related data structures whose relationship with datanodeMap is not that obvious. Plus, it seems that a dedicated function, wipeDatanode(DatanodeID nodeID) is being used to remove a node from datanodeMap. Since this is only for 20 release, ( decommission data states will be changed in next release of HDFS), we decide to only remove the decommission node from the list of dead nodes for displaying purpose only. That is to say, before displaying the list of dead node to either JSP or jmx, we check for decommissioned dead data node and remove them for displaying. We also add a check node.isDecommissioned()) to make sure before removing the data node from dead list, it is already decommissioned.
          Hide
          Tanping Wang added a comment -

          The problem was that after a datanode is registered with a namenode, it is
          kept in the datanodeMap and not removed until it re-registrasted to serve a
          different storageID( e.g. data storage is formatted on this datanode). This
          patch removes the datanode from the datanodeMap after a dfsadmin -refreshNodes
          command, if both of the following two conditions are satisfied:
          1) include list is not empty ( this essentially is a prerequisite.)
          2) datanode does not appear in both include and exclude list.

          This patch is to particularly serve the request from Ops that one
          datanode was in the include list and removing it from both include and
          exclude lists should make the datanode disappear from both live and dead node
          list. The include list is not empty is required. If
          include list is empty, that means all the datanodes are allowed to get registered, even if
          a datanode is not in include and exclude list, it still can get registered
          with the cluster. Therefore, it is kept in the datanodeMap.

          Manually tested with couple different scenarios:
          Scenario I:
          1) include two hosts, H1 and H2 in include file, start cluster
          2) Add H2 to exclude file. Remove H2 from include file, run dfsadmin -refreshNodes
          Result: check web console that H2 disappeared. H1 in live list.

          Scenario II:
          Every steps are the same as scenario I except in step2, still keep H2 in
          include file.
          Result: check web console that H2 disappeared. H1 in live list.

          Scenario III:
          1) leave include and exclude list empty. Start cluster.
          2) add H2 into exclude list, refresh nodes
          2) remove H2 from exclude list, refresh nodes.
          Result: H2 should STILL in the dead node list. Include list is empty, any
          nodes can join. Even remove H2 from exclude list, H2 is not considered to be
          completely removed from the cluster. So we do not remove H2 from datanodeMap. Hence, it appears in dead node list
          still.

          Show
          Tanping Wang added a comment - The problem was that after a datanode is registered with a namenode, it is kept in the datanodeMap and not removed until it re-registrasted to serve a different storageID( e.g. data storage is formatted on this datanode). This patch removes the datanode from the datanodeMap after a dfsadmin -refreshNodes command, if both of the following two conditions are satisfied: 1) include list is not empty ( this essentially is a prerequisite.) 2) datanode does not appear in both include and exclude list. This patch is to particularly serve the request from Ops that one datanode was in the include list and removing it from both include and exclude lists should make the datanode disappear from both live and dead node list. The include list is not empty is required. If include list is empty, that means all the datanodes are allowed to get registered, even if a datanode is not in include and exclude list, it still can get registered with the cluster. Therefore, it is kept in the datanodeMap. Manually tested with couple different scenarios: Scenario I: 1) include two hosts, H1 and H2 in include file, start cluster 2) Add H2 to exclude file. Remove H2 from include file, run dfsadmin -refreshNodes Result: check web console that H2 disappeared. H1 in live list. Scenario II: Every steps are the same as scenario I except in step2, still keep H2 in include file. Result: check web console that H2 disappeared. H1 in live list. Scenario III: 1) leave include and exclude list empty. Start cluster. 2) add H2 into exclude list, refresh nodes 2) remove H2 from exclude list, refresh nodes. Result: H2 should STILL in the dead node list. Include list is empty, any nodes can join. Even remove H2 from exclude list, H2 is not considered to be completely removed from the cluster. So we do not remove H2 from datanodeMap. Hence, it appears in dead node list still.

            People

            • Assignee:
              Tanping Wang
              Reporter:
              Tanping Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development