Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6261

Document for enabling node group layer in HDFS

    Details

    • Type: Task Task
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: documentation
    • Labels:

      Description

      Most of patches from Umbrella JIRA HADOOP-8468 have committed, However there is no site to introduce NodeGroup-aware(HADOOP Virtualization Extensisons) and how to do configuration. so we need to doc it.
      1. Doc NodeGroup-aware relate in http://hadoop.apache.org/docs/current
      2. Doc NodeGroup-aware properties in core-default.xml.

      1. HDFS-6261.010.patch
        110 kB
        Binglin Chang
      2. HDFS-6261.009.patch
        110 kB
        Binglin Chang
      3. HDFS-6261.008.patch
        110 kB
        Binglin Chang
      4. HDFS-6261.007.patch
        110 kB
        Binglin Chang
      5. HDFS-6261.006.patch
        5 kB
        Binglin Chang
      6. HDFS-6261.005.patch
        6 kB
        Binglin Chang
      7. HDFS-6261.004.patch
        6 kB
        Binglin Chang
      8. HDFS-6261.v3.patch
        7 kB
        Wenwu Peng
      9. HDFS-6261.v2.patch
        7 kB
        Binglin Chang
      10. 3-layer-topology.png
        53 kB
        Binglin Chang
      11. 2-layer-topology.png
        30 kB
        Binglin Chang
      12. HDFS-6261.v1.patch
        6 kB
        Binglin Chang
      13. 3layer-topology.png
        10 kB
        Binglin Chang
      14. 4layer-topology.png
        16 kB
        Binglin Chang
      15. HDFS-6261.v1.patch
        6 kB
        Binglin Chang

        Issue Links

          Activity

          Hide
          Junping Du added a comment -

          NodeGroup Topology feature is completed in HDFS part, but haven't finished in YARN (pending on YARN-18 and YARN-19). Agree we can do HDFS first, so move this JIRA to HDFS.

          Show
          Junping Du added a comment - NodeGroup Topology feature is completed in HDFS part, but haven't finished in YARN (pending on YARN-18 and YARN-19 ). Agree we can do HDFS first, so move this JIRA to HDFS.
          Hide
          Binglin Chang added a comment -

          Attach initial patch, the additional png images should be added to:

          hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/3layer-topology.png
          hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/4layer-topology.png

          Show
          Binglin Chang added a comment - Attach initial patch, the additional png images should be added to: hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/3layer-topology.png hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/4layer-topology.png
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641678/3layer-topology.png
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6717//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641678/3layer-topology.png against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6717//console This message is automatically generated.
          Hide
          Binglin Chang added a comment -

          Looks like jenkins used png image file as the patch, so it doesn't apply, resubmit patch

          Show
          Binglin Chang added a comment - Looks like jenkins used png image file as the patch, so it doesn't apply, resubmit patch
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641871/HDFS-6261.v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641871/HDFS-6261.v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6728//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks Binglin Chang for working on this effort.
          I quickly go through it with a few comments below (not completed):

          The original Hadoop topology supports a 3-layer topology looks like following:

          I think it is better to say: previously, Hadoop only support 2-layers topology: rack and host. Mentioning datacenter layer will confuse user as it is never worked even now. For the same reason, we should mention now we support 3 layers topology/locality: rack, nodegroup and host.

          This network topology is designed and work well for Hadoop cluster running on physical server farms. However, for Hadoop running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include:...

          I think the use case of NodeGroup layer is even broader than virtualization and suitable for any sub-dependency of nodes between rack and host layer. So, it could be better to say something like "This network topology is designed to work well on Hadoop cluster that only has rack (switch or power) failure dependency among nodes. However, for other cases, like: Hadoop nodes running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include ..."

          Due to above characteristics in performance and reliability, this layer is not transparent for Hadoop...

          Reliability is more important here, so here better to be "Due to above characteristics in reliability and performance, this layer should't be transparent for Hadoop..."

          1st replica is on the local node or local node group of the writer

          For more precisely, we may say something like: "1st replica is placed on the nearest node to writer in topology. In most cases, it should be on the same node of writer, but could be on other node in the same nodegroup or rack if node of writer is not qualified (i.e. no local datanode or disk is full) to place replica."

          The diagram is better to omit "datacenter" layer according to comments above and red layer of "S1" is better update to "NG1" for reflecting NodeGroup layer.

          Show
          Junping Du added a comment - Thanks Binglin Chang for working on this effort. I quickly go through it with a few comments below (not completed): The original Hadoop topology supports a 3-layer topology looks like following: I think it is better to say: previously, Hadoop only support 2-layers topology: rack and host. Mentioning datacenter layer will confuse user as it is never worked even now. For the same reason, we should mention now we support 3 layers topology/locality: rack, nodegroup and host. This network topology is designed and work well for Hadoop cluster running on physical server farms. However, for Hadoop running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include:... I think the use case of NodeGroup layer is even broader than virtualization and suitable for any sub-dependency of nodes between rack and host layer. So, it could be better to say something like "This network topology is designed to work well on Hadoop cluster that only has rack (switch or power) failure dependency among nodes. However, for other cases, like: Hadoop nodes running on virtualized platform, we have additional "hypervisor" layer, and its characteristics include ..." Due to above characteristics in performance and reliability, this layer is not transparent for Hadoop... Reliability is more important here, so here better to be "Due to above characteristics in reliability and performance, this layer should't be transparent for Hadoop..." 1st replica is on the local node or local node group of the writer For more precisely, we may say something like: "1st replica is placed on the nearest node to writer in topology. In most cases, it should be on the same node of writer, but could be on other node in the same nodegroup or rack if node of writer is not qualified (i.e. no local datanode or disk is full) to place replica." The diagram is better to omit "datacenter" layer according to comments above and red layer of "S1" is better update to "NG1" for reflecting NodeGroup layer.
          Hide
          Binglin Chang added a comment -

          Attach new version of images addressing Junping's comments

          Show
          Binglin Chang added a comment - Attach new version of images addressing Junping's comments
          Hide
          Binglin Chang added a comment -

          Hi Junping, thanks for the review! I updated the patch addressing your comments.

          Show
          Binglin Chang added a comment - Hi Junping, thanks for the review! I updated the patch addressing your comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestDistributedFileSystem

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestDistributedFileSystem +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6753//console This message is automatically generated.
          Hide
          Binglin Chang added a comment -

          The failed test is not caused by this patch, and is tracked in HDFS-5912

          Show
          Binglin Chang added a comment - The failed test is not caused by this patch, and is tracked in HDFS-5912
          Hide
          Binglin Chang added a comment -

          Sorry, looked more carefully, the failure is different, fired HDFS-6308 for this.

          Show
          Binglin Chang added a comment - Sorry, looked more carefully, the failure is different, fired HDFS-6308 for this.
          Hide
          Wenwu Peng added a comment -

          Hi Junping, could you help review again?

          Show
          Wenwu Peng added a comment - Hi Junping, could you help review again?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7605//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642186/HDFS-6261.v2.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7605//console This message is automatically generated.
          Hide
          Wenwu Peng added a comment -

          Re-sync to latest version to solve the patch failure issue.

          Show
          Wenwu Peng added a comment - Re-sync to latest version to solve the patch failure issue.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7607//console This message is automatically generated.
          Hide
          Allen Wittenauer added a comment -

          I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc.

          Show
          Allen Wittenauer added a comment - I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
          against trunk revision 6c3fec5.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9319//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch against trunk revision 6c3fec5. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9319//console This message is automatically generated.
          Hide
          Allen Wittenauer added a comment -

          Cancelling the current patch due to non-applicability and due to comments.

          Show
          Allen Wittenauer added a comment - Cancelling the current patch due to non-applicability and due to comments.
          Hide
          Binglin Chang added a comment -

          I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc

          OK. Will update the patch once HADOOP-11495 is resolved.

          Show
          Binglin Chang added a comment - I'd prefer to see this get merged into the RackAwareness documentation rather than building a completely new doc OK. Will update the patch once HADOOP-11495 is resolved.
          Hide
          Junping Du added a comment -

          Sounds like a reasonable plan, Binglin Chang! I may need more time for HADOOP-11495 because some other JIRA and efforts, please feel free to take that one if you can do it now and I will help to review it. Thanks Allen Wittenauer for comments also!

          Show
          Junping Du added a comment - Sounds like a reasonable plan, Binglin Chang ! I may need more time for HADOOP-11495 because some other JIRA and efforts, please feel free to take that one if you can do it now and I will help to review it. Thanks Allen Wittenauer for comments also!
          Hide
          Junping Du added a comment -

          Hi Binglin Chang, given HADOOP-11495 is already get commit in, would you mind to update your patch here? I will give it a review. Thx!

          Show
          Junping Du added a comment - Hi Binglin Chang , given HADOOP-11495 is already get commit in, would you mind to update your patch here? I will give it a review. Thx!
          Hide
          Binglin Chang added a comment -

          Sorry for the late, will update the patch soon.

          Show
          Binglin Chang added a comment - Sorry for the late, will update the patch soon.
          Hide
          Lars Francke added a comment -

          Hey Binglin Chang just a friendly ping/reminder. Having this documented would be great, thank you!

          Show
          Lars Francke added a comment - Hey Binglin Chang just a friendly ping/reminder. Having this documented would be great, thank you!
          Hide
          Binglin Chang added a comment -

          Sorry... will see if I can get this done this weekend.

          Show
          Binglin Chang added a comment - Sorry... will see if I can get this done this weekend.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 1s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch
          Optional Tests site
          git revision trunk / 446d515
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11119/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 1s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12660927/HDFS-6261.v3.patch Optional Tests site git revision trunk / 446d515 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11119/console This message was automatically generated.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735103/HDFS-6261.004.patch
          Optional Tests site
          git revision trunk / 446d515
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11120/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735103/HDFS-6261.004.patch Optional Tests site git revision trunk / 446d515 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11120/console This message was automatically generated.
          Hide
          Binglin Chang added a comment -

          Remove binary file in patch,
          the 2 png images(2-layer-topology.png, 3-layer-topology.png) should put into hadoop-common-project/hadoop-common/src/site/resources/images/

          Show
          Binglin Chang added a comment - Remove binary file in patch, the 2 png images(2-layer-topology.png, 3-layer-topology.png) should put into hadoop-common-project/hadoop-common/src/site/resources/images/
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735106/HDFS-6261.005.patch
          Optional Tests site
          git revision trunk / 446d515
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11121/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735106/HDFS-6261.005.patch Optional Tests site git revision trunk / 446d515 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11121/console This message was automatically generated.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735109/HDFS-6261.006.patch
          Optional Tests site
          git revision trunk / 446d515
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11122/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735109/HDFS-6261.006.patch Optional Tests site git revision trunk / 446d515 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11122/console This message was automatically generated.
          Hide
          Allen Wittenauer added a comment -

          You can include binary files with 'git format-patch'.

          Show
          Allen Wittenauer added a comment - You can include binary files with 'git format-patch'.
          Hide
          Binglin Chang added a comment -

          Thanks Allen. Update the patch.

          Show
          Binglin Chang added a comment - Thanks Allen. Update the patch.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 58s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 56s Site still builds.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
              6m 19s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735113/HDFS-6261.007.patch
          Optional Tests site
          git revision trunk / 446d515
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11123/artifact/patchprocess/whitespace.txt
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11123/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 site 2m 56s Site still builds. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.     6m 19s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735113/HDFS-6261.007.patch Optional Tests site git revision trunk / 446d515 whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11123/artifact/patchprocess/whitespace.txt Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11123/console This message was automatically generated.
          Hide
          Allen Wittenauer added a comment - - edited

          Awesome! Doc patches are great!

          Now for the review. This isn't comprehensive, but here's a first pass at least.

          A common problem is a missing 'the' in front of 'following'. I pointed it out in a few places, but there are more. Articles in English are tricky. 'Following' is particularly tricky though because 'following' without 'the' or 'a' in front of it is a verb (e.g., "The dog was following the boy" vs. "Hadoop has a following" or "See the following list of cool places to eat".). But hey, at least we don't have genders like the European languages though!

          However, for
          +other cases, like: Hadoop nodes running on virtualized platform, we have
          +additional "hypervisor" layer, and its characteristics include:
          

          I don't know how to parse this phrasing. It feels awkward. I'd probably rewrite as:

          However for some cases, this is insufficient. Take for example Hadoop nodes running on a virtualized platform where there is an additional hypervisor layer. It has the following characteristics:

          +-   The communication price between VMs within the same hypervisor is lower
          +than across hypervisor (physical host) which will have higher throughput,
          +lower latency, and not generating physical network traffic.
          

          Same sort of problem. I'd probably rephrase a bit:

          "The communication price between multiple VMs running on one physical host is lower than the communication price between processes on multiple physical hosts. In addition to the multiple VMs having higher throughput and lower latency between themselves, they do not generate any network traffic on the wire."

          transparent for Hadoop, so
          

          'for' should be 'to'. Hadoop (period). (new sentence) So

          like following:
          

          like the following:

          layer, following polices
          +in hdfs are refined:
          

          the following. HDFS.

          +-   Replica placement policy
          

          I have a feeling bullet points in front of all the items listed under this section may render better. I need to play with it though.

          of writer,
          

          of the writer

          on other
          +    node
          

          on another node

          if node of writer
          

          if the node of the writer

          The remaining replicas are placed randomly across rack and node group to
          +    meet minimum restriction.
          

          I'm confused by this since there are missing articles and/or plurals here. Does this mean randomly across the remaining racks or randomly across all racks including the writer's rack?

          At node level
          

          At the node level

          At block level
          

          At the block level

          Reliability: By never placing more than one replicas on the same node
          +group(physical host), in case of node group failure, only one replica is
          +lost at maximum.
          

          Awkward phrasing. I'd probably rewrite as:

          "Reliability: By never placing more than one replica in the same node
          group (aka physical host), only one replica is lost at maximum in case of node group failure."

          rather than remote node
          

          than a remote

          +3-layer topology tends to support different failure and locality topologies
          +which is primarily driven from the perspective of virtualization, however,
          +it is also possible to use the feature support other scenarios, such as
          +those relating to failures of power supplies, arbitrary sets of physical
          +servers, or collections of servers from same hardware purchase cycle.
          

          This paragraph feels like it should be up closer to the top of these changes.

          Show
          Allen Wittenauer added a comment - - edited Awesome! Doc patches are great! Now for the review. This isn't comprehensive, but here's a first pass at least. A common problem is a missing 'the' in front of 'following'. I pointed it out in a few places, but there are more. Articles in English are tricky. 'Following' is particularly tricky though because 'following' without 'the' or 'a' in front of it is a verb (e.g., "The dog was following the boy" vs. "Hadoop has a following" or "See the following list of cool places to eat".). But hey, at least we don't have genders like the European languages though! However, for +other cases, like: Hadoop nodes running on virtualized platform, we have +additional "hypervisor" layer, and its characteristics include: I don't know how to parse this phrasing. It feels awkward. I'd probably rewrite as: However for some cases, this is insufficient. Take for example Hadoop nodes running on a virtualized platform where there is an additional hypervisor layer. It has the following characteristics: +- The communication price between VMs within the same hypervisor is lower +than across hypervisor (physical host) which will have higher throughput, +lower latency, and not generating physical network traffic. Same sort of problem. I'd probably rephrase a bit: "The communication price between multiple VMs running on one physical host is lower than the communication price between processes on multiple physical hosts. In addition to the multiple VMs having higher throughput and lower latency between themselves, they do not generate any network traffic on the wire." transparent for Hadoop, so 'for' should be 'to'. Hadoop (period). (new sentence) So like following: like the following: layer, following polices +in hdfs are refined: the following. HDFS. +- Replica placement policy I have a feeling bullet points in front of all the items listed under this section may render better. I need to play with it though. of writer, of the writer on other + node on another node if node of writer if the node of the writer The remaining replicas are placed randomly across rack and node group to + meet minimum restriction. I'm confused by this since there are missing articles and/or plurals here. Does this mean randomly across the remaining racks or randomly across all racks including the writer's rack? At node level At the node level At block level At the block level Reliability: By never placing more than one replicas on the same node +group(physical host), in case of node group failure, only one replica is +lost at maximum. Awkward phrasing. I'd probably rewrite as: "Reliability: By never placing more than one replica in the same node group (aka physical host), only one replica is lost at maximum in case of node group failure." rather than remote node than a remote +3-layer topology tends to support different failure and locality topologies +which is primarily driven from the perspective of virtualization, however, +it is also possible to use the feature support other scenarios, such as +those relating to failures of power supplies, arbitrary sets of physical +servers, or collections of servers from same hardware purchase cycle. This paragraph feels like it should be up closer to the top of these changes.
          Hide
          Allen Wittenauer added a comment -

          I have a feeling bullet points in front of all the items listed under this section may render better. I need to play with it though.

          Meh. Ignore that. Finally got the rendered page up and I think it looks fine.

          Show
          Allen Wittenauer added a comment - I have a feeling bullet points in front of all the items listed under this section may render better. I need to play with it though. Meh. Ignore that. Finally got the rendered page up and I think it looks fine.
          Hide
          Binglin Chang added a comment -

          Thanks for the detailed review, nice comments. I made some modifications according to your comments.

          Show
          Binglin Chang added a comment - Thanks for the detailed review, nice comments. I made some modifications according to your comments.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 4m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 58s Site still builds.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
              7m 36s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736515/HDFS-6261.008.patch
          Optional Tests site
          git revision trunk / 5cc3fce
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11181/artifact/patchprocess/whitespace.txt
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11181/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 4m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 site 2m 58s Site still builds. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.     7m 36s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736515/HDFS-6261.008.patch Optional Tests site git revision trunk / 5cc3fce whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/11181/artifact/patchprocess/whitespace.txt Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11181/console This message was automatically generated.
          Hide
          Binglin Chang added a comment -

          remove tailing whitespace

          Show
          Binglin Chang added a comment - remove tailing whitespace
          Hide
          Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 56s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 0s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 20s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736534/HDFS-6261.009.patch
          Optional Tests site
          git revision trunk / 5cc3fce
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11183/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 3m 0s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736534/HDFS-6261.009.patch Optional Tests site git revision trunk / 5cc3fce Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11183/console This message was automatically generated.
          Hide
          Junping Du added a comment -

          Thanks Binglin Chang for updating the patch and Allen Wittenauer for review comments!
          The latest patch get much closer, some additional comments:

          +<property>
          +  <name>net.topology.nodegroup.aware</name>
          +  <value>true</value>
          +</property>
          

          This property is only used in branch-1 (for identify jobtracker's awareness of nodegroup), but not used in trunk and branch-2. Let's remove it.

          +  <name>dfs.block.replicator.classname</name>
          +  <value>
          +    org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyWithNodeGroup
          +  </value>
          

          The package name for BlockPlacementPolicyWithNodeGroup is already be refactor to "org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup". Let's do the update.

          +    The remaining replicas are placed randomly across other nodes.
          

          Better to mention that this random won't conflict policy no. 1: no two replicas will be placed among nodes within the same nodegroup.

          Other looks fine to me.

          Show
          Junping Du added a comment - Thanks Binglin Chang for updating the patch and Allen Wittenauer for review comments! The latest patch get much closer, some additional comments: +<property> + <name>net.topology.nodegroup.aware</name> + <value> true </value> +</property> This property is only used in branch-1 (for identify jobtracker's awareness of nodegroup), but not used in trunk and branch-2. Let's remove it. + <name>dfs.block.replicator.classname</name> + <value> + org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicyWithNodeGroup + </value> The package name for BlockPlacementPolicyWithNodeGroup is already be refactor to "org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithNodeGroup". Let's do the update. + The remaining replicas are placed randomly across other nodes. Better to mention that this random won't conflict policy no. 1: no two replicas will be placed among nodes within the same nodegroup. Other looks fine to me.
          Hide
          Binglin Chang added a comment -

          Thanks for the review Junping!
          I change "nodes" to "node groups", that should have not conflict with "No duplicated replicas are on the same node or node group", and still simple enough to avoid misunderstanding?

          The remaining replicas are placed randomly across other node groups
          
          Show
          Binglin Chang added a comment - Thanks for the review Junping! I change "nodes" to "node groups", that should have not conflict with "No duplicated replicas are on the same node or node group", and still simple enough to avoid misunderstanding? The remaining replicas are placed randomly across other node groups
          Hide
          Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 3m 47s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 57s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              7m 7s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738290/HDFS-6261.010.patch
          Optional Tests site
          git revision trunk / a6cb489
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11264/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 57s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     7m 7s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738290/HDFS-6261.010.patch Optional Tests site git revision trunk / a6cb489 Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11264/console This message was automatically generated.

            People

            • Assignee:
              Binglin Chang
              Reporter:
              Wenwu Peng
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development