Hadoop Common
  1. Hadoop Common
  2. HADOOP-9451

Node with one topology layer should be handled as fault topology when NodeGroup layer is enabled

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.2
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: net
    • Labels:
      None

      Description

      Currently, nodes with one layer topology are allowed to join in the cluster that with enabling NodeGroup layer which cause some exception cases.
      When NodeGroup layer is enabled, the cluster should assumes that at least two layer (Rack/NodeGroup) is valid topology for each nodes, so should throw exceptions for one layer node in joining.

      1. HADOOP-9451-branch-1.patch
        2 kB
        Junping Du
      2. HADOOP-9451-v2.patch
        9 kB
        Junping Du
      3. HADOOP-9451.patch
        16 kB
        Junping Du
      4. HDFS-4652-branch1.patch
        5 kB
        Junping Du
      5. HDFS-4652.patch
        5 kB
        Junping Du

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12576046/HDFS-4652.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestFcHdfsSymlink

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4166//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4166//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576046/HDFS-4652.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFcHdfsSymlink +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4166//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4166//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          The UT test failed seems unrelated.
          Attach branch-1 patch as well.

          Show
          Junping Du added a comment - The UT test failed seems unrelated. Attach branch-1 patch as well.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12576072/HDFS-4652-branch1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4167//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576072/HDFS-4652-branch1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4167//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          The UT failed is a non-related issue as previous Jenkins build with the same failure. Already filed HDFS-4653 to address this.

          Show
          Junping Du added a comment - The UT failed is a non-related issue as previous Jenkins build with the same failure. Already filed HDFS-4653 to address this.
          Hide
          Luke Lu added a comment -

          The patches lgtm. +1. Will commit shortly.

          Show
          Luke Lu added a comment - The patches lgtm. +1. Will commit shortly.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12576072/HDFS-4652-branch1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2401//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576072/HDFS-4652-branch1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2401//console This message is automatically generated.
          Hide
          Luke Lu added a comment -

          Sorry forgot about the rule to avoid changes across projects. Moved to common, as the patch is for common NetworkTopologyWithNodeGroup, though the test is in hdfs. I think the "unit" test should be in common as well. It'll be great if you can submit a patch for common only.

          Show
          Luke Lu added a comment - Sorry forgot about the rule to avoid changes across projects. Moved to common, as the patch is for common NetworkTopologyWithNodeGroup, though the test is in hdfs. I think the "unit" test should be in common as well. It'll be great if you can submit a patch for common only.
          Hide
          Luke Lu added a comment -

          branch-1 changes is acceptable.

          Show
          Luke Lu added a comment - branch-1 changes is acceptable.
          Hide
          Junping Du added a comment -

          Thanks Luke for review and comments. I thought unit test should be in common too. But TestNetworkTopology or TestNetworkTopologyWithNodeGroup are all in HDFS project now. Shall we move them to common?

          Show
          Junping Du added a comment - Thanks Luke for review and comments. I thought unit test should be in common too. But TestNetworkTopology or TestNetworkTopologyWithNodeGroup are all in HDFS project now. Shall we move them to common?
          Hide
          Luke Lu added a comment -

          Some of these test are actually more like "integration" tests (especially the ones that use MiniDFSCluster). I suggest that you just move the TestNetworkTopologyWithNodeGroup as it has minimal dependency on hdfs stuff.

          Show
          Luke Lu added a comment - Some of these test are actually more like "integration" tests (especially the ones that use MiniDFSCluster). I suggest that you just move the TestNetworkTopologyWithNodeGroup as it has minimal dependency on hdfs stuff.
          Hide
          Junping Du added a comment -

          Yes. There are minimal dependency on hdfs stuff but could be replaced (DataNodeDescriptor => NodeBase) in test case. Move TestNetworkTopologyWithNodeGroup to common project to stay together with NetworkTopologyWithNodeGroup. We can consider to move TestNetworkTopology to common project too (the same minimal changes).

          Show
          Junping Du added a comment - Yes. There are minimal dependency on hdfs stuff but could be replaced (DataNodeDescriptor => NodeBase) in test case. Move TestNetworkTopologyWithNodeGroup to common project to stay together with NetworkTopologyWithNodeGroup. We can consider to move TestNetworkTopology to common project too (the same minimal changes).
          Hide
          Luke Lu added a comment -

          Some tests in TestNetworkTopology uses MiniDFSCluster, so you'll need to split the test suite into two. Feel free to provide two patches (two jiras as well, one for common one for HDFS)

          Show
          Luke Lu added a comment - Some tests in TestNetworkTopology uses MiniDFSCluster, so you'll need to split the test suite into two. Feel free to provide two patches (two jiras as well, one for common one for HDFS)
          Hide
          Luke Lu added a comment -

          Please move the deletion of the TestNetworkTopologyWithNodeGroup from hdfs project to HDFS-4665.

          Show
          Luke Lu added a comment - Please move the deletion of the TestNetworkTopologyWithNodeGroup from hdfs project to HDFS-4665 .
          Hide
          Luke Lu added a comment -

          It'll be great if you can backport the trunk patch to branch-1 (only needs one patch including the move) as well.

          Show
          Luke Lu added a comment - It'll be great if you can backport the trunk patch to branch-1 (only needs one patch including the move) as well.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12576765/HADOOP-9451.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestFcHdfsSymlink

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2404//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2404//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576765/HADOOP-9451.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestFcHdfsSymlink +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2404//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2404//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Hi Luke, Thanks for comments. I update the patch to contains common part only and put removing TestNetworkTopologyWithNodeGroup in HDFS part in HDFS-4665. Will deliver a branch-1 patch soon.

          Show
          Junping Du added a comment - Hi Luke, Thanks for comments. I update the patch to contains common part only and put removing TestNetworkTopologyWithNodeGroup in HDFS part in HDFS-4665 . Will deliver a branch-1 patch soon.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12576884/HADOOP-9451-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2408//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2408//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576884/HADOOP-9451-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2408//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2408//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Attach branch-1 patch.

          Show
          Junping Du added a comment - Attach branch-1 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12576892/HADOOP-9451-branch-1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2409//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12576892/HADOOP-9451-branch-1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2409//console This message is automatically generated.
          Hide
          Luke Lu added a comment -

          +1 on the latest patches. Committed to trunk, branch-1 and 1.2. Thanks Junping!

          Show
          Luke Lu added a comment - +1 on the latest patches. Committed to trunk, branch-1 and 1.2. Thanks Junping!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3561 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3561/)
          HADOOP-9451. Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3561 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3561/ ) HADOOP-9451 . Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/174/)
          HADOOP-9451. Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #174 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/174/ ) HADOOP-9451 . Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/)
          HADOOP-9451. Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301)

          Result = FAILURE
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1363 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1363/ ) HADOOP-9451 . Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301) Result = FAILURE llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1390 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/)
          HADOOP-9451. Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1390 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1390/ ) HADOOP-9451 . Fault single-layer config if node group topology is enabled. (Junping Du via llu) (Revision 1464301) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1464301 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopologyWithNodeGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetworkTopologyWithNodeGroup.java
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Suresh Srinivas added a comment -

          Luke Lu This needs to be merge to branch-2.1.0 along with other NodeGroup topology code.

          Show
          Suresh Srinivas added a comment - Luke Lu This needs to be merge to branch-2.1.0 along with other NodeGroup topology code.
          Hide
          Luke Lu added a comment -

          Committed to branch-2 and 2.1-beta as well. Thanks for the note, Suresh!

          Show
          Luke Lu added a comment - Committed to branch-2 and 2.1-beta as well. Thanks for the note, Suresh!

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development