Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6261

Add document for enabling node group layer in HDFS

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: documentation
    • Labels:

      Description

      Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it.
      1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
      2. Doc NodeGroup-aware properties in core-default.xml.

      1. 2-layer-topology.png
        30 kB
        Binglin Chang
      2. 3layer-topology.png
        10 kB
        Binglin Chang
      3. 3-layer-topology.png
        53 kB
        Binglin Chang
      4. 4layer-topology.png
        16 kB
        Binglin Chang
      5. HDFS-6261.v1.patch
        6 kB
        Binglin Chang
      6. HDFS-6261.v1.patch
        6 kB
        Binglin Chang
      7. HDFS-6261.v2.patch
        7 kB
        Binglin Chang
      8. HDFS-6261.v3.patch
        7 kB
        Wenwu Peng

        Issue Links

          Activity

          Wenwu Peng created issue -
          Binglin Chang made changes -
          Field Original Value New Value
          Assignee Binglin Chang [ decster ]
          Binglin Chang made changes -
          Summary Doc NodeGroup-aware(HADOOP Virtualization Extensisons) Add document for node group related configs
          Junping Du made changes -
          Summary Add document for node group related configs Add document for enabling node group layer in HDFS
          Junping Du made changes -
          Project Hadoop Common [ 12310240 ] Hadoop HDFS [ 12310942 ]
          Key HADOOP-10497 HDFS-6261
          Fix Version/s site [ 12312854 ]
          Component/s documentation [ 12312918 ]
          Component/s documentation [ 12311160 ]
          Hide
          Junping Du added a comment -

          NodeGroup Topology feature is completed in HDFS part, but haven't finished in YARN (pending on YARN-18 and YARN-19). Agree we can do HDFS first, so move this JIRA to HDFS.

          Show
          Junping Du added a comment - NodeGroup Topology feature is completed in HDFS part, but haven't finished in YARN (pending on YARN-18 and YARN-19 ). Agree we can do HDFS first, so move this JIRA to HDFS.
          Junping Du made changes -
          Link This issue is related to HADOOP-10512 [ HADOOP-10512 ]
          Binglin Chang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Binglin Chang added a comment -

          Attach initial patch, the additional png images should be added to:

          hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/3layer-topology.png
          hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/4layer-topology.png

          Show
          Binglin Chang added a comment - Attach initial patch, the additional png images should be added to: hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/3layer-topology.png hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/4layer-topology.png
          Binglin Chang made changes -
          Attachment HDFS-6261.v1.patch [ 12641676 ]
          Attachment 4layer-topology.png [ 12641677 ]
          Attachment 3layer-topology.png [ 12641678 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641678/3layer-topology.png
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6717//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641678/3layer-topology.png against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6717//console This message is automatically generated.
          Hide
          Binglin Chang added a comment -

          Looks like jenkins used png image file as the patch, so it doesn't apply, resubmit patch

          Show
          Binglin Chang added a comment - Looks like jenkins used png image file as the patch, so it doesn't apply, resubmit patch
          Binglin Chang made changes -
          Attachment HDFS-6261.v1.patch [ 12641871 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641871/HDFS-6261.v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641871/HDFS-6261.v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks Binglin Chang for working on this effort.
          I quickly go through it with a few comments below (not completed):

          The original Hadoop topology supports a 3-layer topology looks like following:

          I think it is better to say: previously, Hadoop only support 2-layers topology: rack and host. Mentioning datacenter layer will confuse user as it is never worked even now. For the same reason, we should mention now we support 3 layers topology/locality: rack, nodegroup and host.

          This network topology is designed and work well for Hadoop cluster running on physical server farms. However, for Hadoop running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include:...

          I think the use case of NodeGroup layer is even broader than virtualization and suitable for any sub-dependency of nodes between rack and host layer. So, it could be better to say something like "This network topology is designed to work well on Hadoop cluster that only has rack (switch or power) failure dependency among nodes. However, for other cases, like: Hadoop nodes running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include ..."

          Due to above characteristics in performance and reliability, this layer is not transparent for Hadoop...

          Reliability is more important here, so here better to be "Due to above characteristics in reliability and performance, this layer should't be transparent for Hadoop..."

          1st replica is on the local node or local node group of the writer

          For more precisely, we may say something like: "1st replica is placed on the nearest node to writer in topology. In most cases, it should be on the same node of writer, but could be on other node in the same nodegroup or rack if node of writer is not qualified (i.e. no local datanode or disk is full) to place replica."

          The diagram is better to omit "datacenter" layer according to comments above and red layer of "S1" is better update to "NG1" for reflecting NodeGroup layer.

          Show
          Junping Du added a comment - Thanks Binglin Chang for working on this effort. I quickly go through it with a few comments below (not completed): The original Hadoop topology supports a 3-layer topology looks like following: I think it is better to say: previously, Hadoop only support 2-layers topology: rack and host. Mentioning datacenter layer will confuse user as it is never worked even now. For the same reason, we should mention now we support 3 layers topology/locality: rack, nodegroup and host. This network topology is designed and work well for Hadoop cluster running on physical server farms. However, for Hadoop running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include:... I think the use case of NodeGroup layer is even broader than virtualization and suitable for any sub-dependency of nodes between rack and host layer. So, it could be better to say something like "This network topology is designed to work well on Hadoop cluster that only has rack (switch or power) failure dependency among nodes. However, for other cases, like: Hadoop nodes running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include ..." Due to above characteristics in performance and reliability, this layer is not transparent for Hadoop... Reliability is more important here, so here better to be "Due to above characteristics in reliability and performance, this layer should't be transparent for Hadoop..." 1st replica is on the local node or local node group of the writer For more precisely, we may say something like: "1st replica is placed on the nearest node to writer in topology. In most cases, it should be on the same node of writer, but could be on other node in the same nodegroup or rack if node of writer is not qualified (i.e. no local datanode or disk is full) to place replica." The diagram is better to omit "datacenter" layer according to comments above and red layer of "S1" is better update to "NG1" for reflecting NodeGroup layer.
          Hide
          Binglin Chang added a comment -

          Attach new version of images addressing Junping's comments

          Show
          Binglin Chang added a comment - Attach new version of images addressing Junping's comments
          Binglin Chang made changes -
          Attachment 2-layer-topology.png [ 12642184 ]
          Attachment 3-layer-topology.png [ 12642185 ]
          Binglin Chang made changes -
          Attachment HDFS-6261.v2.patch [ 12642186 ]
          Hide
          Binglin Chang added a comment -

          Hi Junping, thanks for the review! I updated the patch addressing your comments.

          Show
          Binglin Chang added a comment - Hi Junping, thanks for the review! I updated the patch addressing your comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDistributedFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDistributedFileSystem +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//console This message is automatically generated.
          Hide
          Binglin Chang added a comment -

          The failed test is not caused by this patch, and is tracked in HDFS-5912

          Show
          Binglin Chang added a comment - The failed test is not caused by this patch, and is tracked in HDFS-5912
          Hide
          Binglin Chang added a comment -

          Sorry, looked more carefully, the failure is different, fired HDFS-6308 for this.

          Show
          Binglin Chang added a comment - Sorry, looked more carefully, the failure is different, fired HDFS-6308 for this.
          Hide
          Wenwu Peng added a comment -

          Hi Junping, could you help review again?

          Show
          Wenwu Peng added a comment - Hi Junping, could you help review again?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7605//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7605//console This message is automatically generated.
          Hide
          Wenwu Peng added a comment -

          Re-sync to latest version to solve the patch failure issue.

          Show
          Wenwu Peng added a comment - Re-sync to latest version to solve the patch failure issue.
          Wenwu Peng made changes -
          Attachment HDFS-6261.v3.patch [ 12660927 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//console This message is automatically generated.
          Hide
          Allen Wittenauer added a comment -

          I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc.

          Show
          Allen Wittenauer added a comment - I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
          against trunk revision 6c3fec5.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9319//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch against trunk revision 6c3fec5. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9319//console This message is automatically generated.
          Hide
          Allen Wittenauer added a comment -

          Cancelling the current patch due to non-applicability and due to comments.

          Show
          Allen Wittenauer added a comment - Cancelling the current patch due to non-applicability and due to comments.
          Allen Wittenauer made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Binglin Chang added a comment -

          I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc

          OK. Will update the patch once HADOOP-11495 is resolved.

          Show
          Binglin Chang added a comment - I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc OK. Will update the patch once HADOOP-11495 is resolved.
          Hide
          Junping Du added a comment -

          Sounds like a reasonable plan, Binglin Chang! I may need more time for HADOOP-11495 because some other JIRA and efforts, please feel free to take that one if you can do it now and I will help to review it. Thanks Allen Wittenauer for comments also!

          Show
          Junping Du added a comment - Sounds like a reasonable plan, Binglin Chang ! I may need more time for HADOOP-11495 because some other JIRA and efforts, please feel free to take that one if you can do it now and I will help to review it. Thanks Allen Wittenauer for comments also!

            People

            • Assignee:
              Binglin Chang
              Reporter:
              Wenwu Peng
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development