Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3170

YARN architecture document needs updating

    Details

    • Hadoop Flags:
      Reviewed

      Description

      The marketing paragraph at the top, "NextGen MapReduce", etc are all marketing rather than actual descriptions. It also needs some general updates, esp given it reads as though 0.23 was just released yesterday.

      1. YARN-3170.patch
        2 kB
        Brahma Reddy Battula
      2. YARN-3170-002.patch
        2 kB
        Brahma Reddy Battula
      3. YARN-3170-003.patch
        2 kB
        Brahma Reddy Battula
      4. YARN-3170-004.patch
        2 kB
        Brahma Reddy Battula
      5. YARN-3170-005.patch
        3 kB
        Brahma Reddy Battula
      6. YARN-3170-006.patch
        3 kB
        Brahma Reddy Battula
      7. YARN-3170-007.patch
        4 kB
        Brahma Reddy Battula
      8. YARN-3170-008.patch
        5 kB
        Brahma Reddy Battula
      9. YARN-3170-009.patch
        6 kB
        Brahma Reddy Battula
      10. YARN-3170-010.patch
        6 kB
        Brahma Reddy Battula
      11. YARN-3170-011.patch
        6 kB
        Brahma Reddy Battula

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Hello Allen Wittenauer

          I want to update like following

          Apache Hadoop NextGen MapReduce (YARN)

          Apache Hadoop Yarn

          MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.

          will remove this line

          The fundamental idea of MRv2

          will update YARN instead of MRv2.

          Please give your inputs

          Show
          brahmareddy Brahma Reddy Battula added a comment - Hello Allen Wittenauer I want to update like following Apache Hadoop NextGen MapReduce (YARN) Apache Hadoop Yarn MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN. will remove this line The fundamental idea of MRv2 will update YARN instead of MRv2. Please give your inputs
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Allen Wittenauer can I go head like above..? please give your inputs ..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Allen Wittenauer can I go head like above..? please give your inputs ..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Allen Wittenauer Kindly review attached patch.. thanks..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Allen Wittenauer Kindly review attached patch.. thanks..
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12726444/YARN-3170.patch
          against trunk revision 5459b24.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7392//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7392//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726444/YARN-3170.patch against trunk revision 5459b24. +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7392//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7392//console This message is automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Brahma Reddy Battula thank you for taking this issue.

          +Apache Hadoop Yarn
          

          Yarn should be YARN here. +1 after the comment is addressed.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Brahma Reddy Battula thank you for taking this issue. +Apache Hadoop Yarn Yarn should be YARN here. +1 after the comment is addressed.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsuyoshi Ozawa thanks a lot for taking look into this issue..Updated the patch based on your comment..Kindly Review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsuyoshi Ozawa thanks a lot for taking look into this issue..Updated the patch based on your comment..Kindly Review..
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Brahma Reddy Battula I found some additional points - could you check them?

          I think we need a lead sentence as a introduction. My suggestion is: "The architecture of Hadoop has been changed in Hadoop 2. Hadoop/MapReduce runs on YARN(Yet Another Resource Negotiator), more generic resource management framework. We call MapReduce running on YARN "MapReduce 2.0 (MRv2)"."

          -MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN.
          

          A following line shouldn't be modified since MRv2 has combatibility with hadoop-1.x.

          -MRV2 maintains **API compatibility** with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile.
          

          What do you think?

          Show
          ozawa Tsuyoshi Ozawa added a comment - Brahma Reddy Battula I found some additional points - could you check them? I think we need a lead sentence as a introduction. My suggestion is: "The architecture of Hadoop has been changed in Hadoop 2. Hadoop/MapReduce runs on YARN(Yet Another Resource Negotiator), more generic resource management framework. We call MapReduce running on YARN "MapReduce 2.0 (MRv2)"." -MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN. A following line shouldn't be modified since MRv2 has combatibility with hadoop-1.x. -MRV2 maintains **API compatibility** with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile. What do you think?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsuyoshi Ozawa I agree with you, it's make sense.. Updated the patch..Kindly Review..Thanks

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsuyoshi Ozawa I agree with you, it's make sense.. Updated the patch..Kindly Review..Thanks
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 50s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 54s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 10s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731478/YARN-3170-003.patch
          Optional Tests site
          git revision trunk / a2d40bc
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7813/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 site 2m 54s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 10s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731478/YARN-3170-003.patch Optional Tests site git revision trunk / a2d40bc Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7813/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Brahma Reddy Battula thank you for updating.

          We call MapReduce running on YARN "MapReduce 2.0 (MRv2).

          A trailing double quotation is missing. Please add it before the period.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Brahma Reddy Battula thank you for updating. We call MapReduce running on YARN "MapReduce 2.0 (MRv2). A trailing double quotation is missing. Please add it before the period.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsuyoshi Ozawa updated the patch..Kindly Review..thanks

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsuyoshi Ozawa updated the patch..Kindly Review..thanks
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 53s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 57s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 13s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731885/YARN-3170-004.patch
          Optional Tests site
          git revision trunk / 3fa2efc
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7860/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 57s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 13s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731885/YARN-3170-004.patch Optional Tests site git revision trunk / 3fa2efc Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7860/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          I'll be honest: I greatly dislike that entire first paragraph. MRv2 needs to be struck from the vocabulary. It was a marketing ploy to get YARN's acceptance into Hadoop as a subproject. It also helps underscore the problems of what to call the "new" MR API in the actual MR subproject. I'm inclined to think the entire paragraph should just get deleted.

          The second paragraph should be rewritten. There's little value in comparing YARN to earlier versions of Hadoop at this point. Don't describe YARN in terms of the JobTracker. If I'm new to Hadoop, I have no idea what the heck a JT even is.

           An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs.
          • Drop "in the classical sense of Map-Reduce jobs".
           The ResourceManager and per-node slave, the NodeManager (*NM*), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system.
          
          • Drop "per-node slave,"
          • Drop "(NM)"
          • Add a description of the node manager after the description of the resource manager in this paragraph.
          Show
          aw Allen Wittenauer added a comment - I'll be honest: I greatly dislike that entire first paragraph. MRv2 needs to be struck from the vocabulary. It was a marketing ploy to get YARN's acceptance into Hadoop as a subproject. It also helps underscore the problems of what to call the "new" MR API in the actual MR subproject. I'm inclined to think the entire paragraph should just get deleted. The second paragraph should be rewritten. There's little value in comparing YARN to earlier versions of Hadoop at this point. Don't describe YARN in terms of the JobTracker. If I'm new to Hadoop, I have no idea what the heck a JT even is. An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. Drop "in the classical sense of Map-Reduce jobs". The ResourceManager and per-node slave, the NodeManager (*NM*), form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. Drop "per-node slave," Drop "( NM )" Add a description of the node manager after the description of the resource manager in this paragraph.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Allen Wittenauer Thanks for taking look into this issue.. Updated the patch based on your comments..Kindly review...Let me anyother rework in second paragraph ( Mainly first line )...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Allen Wittenauer Thanks for taking look into this issue.. Updated the patch based on your comments..Kindly review...Let me anyother rework in second paragraph ( Mainly first line )...
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 53s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 55s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 11s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12731957/YARN-3170-005.patch
          Optional Tests site
          git revision trunk / b9cebfc
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7865/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 55s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 11s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731957/YARN-3170-005.patch Optional Tests site git revision trunk / b9cebfc Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7865/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Allen Wittenauer's suggestion makes sense to me, and The latest patch looks good to me. +1, checking this in.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Allen Wittenauer 's suggestion makes sense to me, and The latest patch looks good to me. +1, checking this in.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Brahma Reddy Battula, I found the last line includes the term "MRv2".

          MRV2 maintains **API compatibility** with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile.
          

          My suggestion is as follows:

          • MapReduce in hadoop-2.x maintains *API compatibility* ~
          • This means that all Map-Reduce jobs should still run unchanged on top of YARN with just a recompile.

          I appreciate if you double check the reviwer's comment, though I sometimes do mistakes

          Show
          ozawa Tsuyoshi Ozawa added a comment - Brahma Reddy Battula , I found the last line includes the term "MRv2". MRV2 maintains **API compatibility** with previous stable release (hadoop-1.x). This means that all Map-Reduce jobs should still run unchanged on top of MRv2 with just a recompile. My suggestion is as follows: MapReduce in hadoop-2.x maintains * API compatibility * ~ This means that all Map-Reduce jobs should still run unchanged on top of YARN with just a recompile. I appreciate if you double check the reviwer's comment, though I sometimes do mistakes
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Tsuyoshi Ozawa for reminding .. After allen comments, Yes I missed this... Kindly review the latest patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Tsuyoshi Ozawa for reminding .. After allen comments, Yes I missed this... Kindly review the latest patch..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Tsuyoshi Ozawa for reminding .. After allen comments, Yes I missed this... Kindly review the latest patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Tsuyoshi Ozawa for reminding .. After allen comments, Yes I missed this... Kindly review the latest patch..
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 51s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 56s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 11s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732566/YARN-3170-006.patch
          Optional Tests site
          git revision trunk / 065d8f2
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7918/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 51s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 56s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 11s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732566/YARN-3170-006.patch Optional Tests site git revision trunk / 065d8f2 Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7918/console This message was automatically generated.
          Hide
          aw Allen Wittenauer added a comment -
          split up the two major functionalities such as resource management
          

          ... split up the functionalities of resource management ...

          NodeManager , form 
          

          Drop the comma.

          system.The NodeManager 
          

          Spacing.

           The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure.
           
          -The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler.
          -
           The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress.
          
          

          With this delete, these paragraphs should probably be merged.

           on top of MRv2 with
          

          s,MRv2,YARN,

          Show
          aw Allen Wittenauer added a comment - split up the two major functionalities such as resource management ... split up the functionalities of resource management ... NodeManager , form Drop the comma. system.The NodeManager Spacing. The ApplicationsManager is responsible for accepting job-submissions, negotiating the first container for executing the application specific ApplicationMaster and provides the service for restarting the ApplicationMaster container on failure. -The NodeManager is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler. - The per-application ApplicationMaster has the responsibility of negotiating appropriate resource containers from the Scheduler, tracking their status and monitoring for progress. With this delete, these paragraphs should probably be merged. on top of MRv2 with s,MRv2,YARN,
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Allen Wittenauer thanks for review..Updated patch based on your comments..Kindly review!!

          Show
          brahmareddy Brahma Reddy Battula added a comment - Allen Wittenauer thanks for review..Updated patch based on your comments..Kindly review!!
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 58s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 59s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 21s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732694/YARN-3170-007.patch
          Optional Tests site
          git revision trunk / 281d47a
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7926/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 59s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732694/YARN-3170-007.patch Optional Tests site git revision trunk / 281d47a Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7926/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          One additional comment from me:

          This means that all Map-Reduce jobs should still run unchanged on top of YARN with just a recompile.

          s/Map-Reduce/MapReduce/

          Show
          ozawa Tsuyoshi Ozawa added a comment - One additional comment from me: This means that all Map-Reduce jobs should still run unchanged on top of YARN with just a recompile. s/Map-Reduce/MapReduce/
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Tsuyoshi Ozawa Updated the patch to address your comment..Kindly review.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Tsuyoshi Ozawa Updated the patch to address your comment..Kindly review.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 2s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 17s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732784/YARN-3170-008.patch
          Optional Tests site
          git revision trunk / b2c85db
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7931/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 3m 2s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 17s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732784/YARN-3170-008.patch Optional Tests site git revision trunk / b2c85db Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7931/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          LGTM. Allen Wittenauer please commit it if you are okay.

          Show
          ozawa Tsuyoshi Ozawa added a comment - LGTM. Allen Wittenauer please commit it if you are okay.
          Hide
          aw Allen Wittenauer added a comment -
          and job scheduling/monitoring, 
          

          Unnecessary comma.

          The Scheduler has a pluggable policy plug-in,
          

          Awkward phrasing. Unnecessary comma.

          In the first version, only memory is supported.
          

          Remove. It isn't true anymore and highlights one of the things I absolutely hate about Hadoop documentation: roadmaps don't belong here.

          The current MapReduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in.
          

          "MapReduce" should be removed. "of plug-ins"; no the & make it plural. Also link to their respective documentation pages.

          The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources
          

          Drop this sentence.

          Show
          aw Allen Wittenauer added a comment - and job scheduling/monitoring, Unnecessary comma. The Scheduler has a pluggable policy plug-in, Awkward phrasing. Unnecessary comma. In the first version, only memory is supported. Remove. It isn't true anymore and highlights one of the things I absolutely hate about Hadoop documentation: roadmaps don't belong here. The current MapReduce schedulers such as the CapacityScheduler and the FairScheduler would be some examples of the plug-in. "MapReduce" should be removed. "of plug-ins"; no the & make it plural. Also link to their respective documentation pages. The CapacityScheduler supports hierarchical queues to allow for more predictable sharing of cluster resources Drop this sentence.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 54s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 55s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 13s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736559/YARN-3170-009.patch
          Optional Tests site
          git revision trunk / 63e3fee
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8150/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 54s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 55s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 13s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736559/YARN-3170-009.patch Optional Tests site git revision trunk / 63e3fee Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8150/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Allen Wittenauer thanks a lot for your comments.. Updated the patch based on your comments..Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Allen Wittenauer thanks a lot for your comments.. Updated the patch based on your comments..Kindly review..
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 3m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 55s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 20s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736570/YARN-3170-009.patch
          Optional Tests site
          git revision trunk / 63e3fee
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8151/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 55s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736570/YARN-3170-009.patch Optional Tests site git revision trunk / 63e3fee Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8151/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          The Scheduler has a pluggable policy plug-in

          I think Allen means the sentence is awkward since "pluggable" "plug-in" sounds redundant. Could you fix it?

          Show
          ozawa Tsuyoshi Ozawa added a comment - The Scheduler has a pluggable policy plug-in I think Allen means the sentence is awkward since "pluggable" "plug-in" sounds redundant. Could you fix it?
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 2m 58s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 2m 55s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 18s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736746/YARN-3170-010.patch
          Optional Tests site
          git revision trunk / 990078b
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8162/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 2m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 2m 55s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 18s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736746/YARN-3170-010.patch Optional Tests site git revision trunk / 990078b Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8162/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Updated patch..Kindly review!!

          Show
          brahmareddy Brahma Reddy Battula added a comment - Updated patch..Kindly review!!
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          +1, checking this in.

          Show
          ozawa Tsuyoshi Ozawa added a comment - +1, checking this in.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Brahma Reddy Battula Oh, the change against line 1 breaks the comment. Could you fix it?

          <!--
          +!---

          Show
          ozawa Tsuyoshi Ozawa added a comment - Brahma Reddy Battula Oh, the change against line 1 breaks the comment. Could you fix it? <! -- +!---
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Oh.. Tsuyoshi Ozawa thanks a lot for review.. Updated the patch..Kindly Review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Oh.. Tsuyoshi Ozawa thanks a lot for review.. Updated the patch..Kindly Review..
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 3m 16s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 site 3m 0s Site still builds.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              6m 39s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12745248/YARN-3170-011.patch
          Optional Tests site
          git revision trunk / 4084eaf
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8531/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 3m 16s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 site 3m 0s Site still builds. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     6m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745248/YARN-3170-011.patch Optional Tests site git revision trunk / 4084eaf Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8531/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          +1

          Show
          ozawa Tsuyoshi Ozawa added a comment - +1
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Committed this to trunk, branch-2, and branch-2.7. Thanks Brahma Reddy Battula for your contribution and thanks Allen Wittenauer for your iterative reviews!

          Show
          ozawa Tsuyoshi Ozawa added a comment - Committed this to trunk, branch-2, and branch-2.7. Thanks Brahma Reddy Battula for your contribution and thanks Allen Wittenauer for your iterative reviews!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8168 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8168/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8168 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8168/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #257 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/257/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #257 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/257/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #987 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/987/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #987 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/987/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2184/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2184/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/245/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/245/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/255/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #255 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/255/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2203/)
          YARN-3170. YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2203/ ) YARN-3170 . YARN architecture document needs updating. Contirubted by Brahma Reddy Battula. (ozawa: rev edcaae44c10b7e88e68fa97afd32e4da4a9d8df7) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YARN.md hadoop-yarn-project/CHANGES.txt

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development