Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5646

Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None

      Description

      This is for adding documentation regarding the scheduling of OPPORTUNISTIC containers.
      It includes both the centralized (YARN-5220) and the distributed (YARN-2877) scheduling.

      1. YARN-5646.addendum.patch
        0.7 kB
        Arun Suresh
      2. YARN-5646.004.patch
        53 kB
        Konstantinos Karanasos
      3. YARN-5646.003.patch
        52 kB
        Konstantinos Karanasos
      4. YARN-5646.002.patch
        49 kB
        Konstantinos Karanasos
      5. YARN-5646.001.patch
        28 kB
        Konstantinos Karanasos

        Issue Links

          Activity

          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Attaching documentation.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Attaching documentation.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Please let's wait for a few days before committing this.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Please let's wait for a few days before committing this.
          Hide
          asuresh Arun Suresh added a comment -

          Karthik Kambatla, Wangda Tan, Jian He.. do let us know what you think of the documentation.

          Show
          asuresh Arun Suresh added a comment - Karthik Kambatla , Wangda Tan , Jian He .. do let us know what you think of the documentation.
          Hide
          asuresh Arun Suresh added a comment -
          Show
          asuresh Arun Suresh added a comment - Subru Krishnan too..
          Hide
          asuresh Arun Suresh added a comment -

          I like the future work part, but don't know if we should have that section...
          This is more a manual for usage for end users... not sure if it must be linked to JIRA tickets.

          Show
          asuresh Arun Suresh added a comment - I like the future work part, but don't know if we should have that section... This is more a manual for usage for end users... not sure if it must be linked to JIRA tickets.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          I understand the concern about future work. I added it on purpose, so that people that read the documentation, can get an idea of open items (and even contribute to them).
          But if you all think it's not suitable, I can remove it.

          Karthik Kambatla, I also included the motivation for over-commitment through opportunistic containers, but made clear in the text that we do not yet support it. Once over-commitment is also available, we will update the document.

          Show
          kkaranasos Konstantinos Karanasos added a comment - I understand the concern about future work. I added it on purpose, so that people that read the documentation, can get an idea of open items (and even contribute to them). But if you all think it's not suitable, I can remove it. Karthik Kambatla , I also included the motivation for over-commitment through opportunistic containers, but made clear in the text that we do not yet support it. Once over-commitment is also available, we will update the document.
          Hide
          templedf Daniel Templeton added a comment -

          Overall a well written doc. Minor comments:

          • "list loaded nodes" should be "least loaded nodes"
          • "In case of" should be "In the case of"
          • "the NM on a node whose queue length is above the threshold, discards opportunistic containers to meet this maximal value" should drop the comma and be "an NM ..."
          • It would be good to call out the defaults for the properties
          • It's not obvious what the incremental properties do. Min and max are self-explanatory, but I think you have to explain the increments.
          • yarn.opportunistic-container-allocation.nodes-used needs more explanation. I can guess what it does, but it would be better if you just explain it more.
          • yarn.nm-container-queuing.min-queue-length needs more explanation. I have no idea what it does from the doc. How can a minimum be enforced? What if there are no jobs?
          • Same for yarn.nm-container-queuing.min-queue-wait-time-ms
          • "if a map task fail" should be "if a map task fails"
          • "Also, when clicking" should drop the "Also"
          • "the open JIRAs" should just be "the JIRAs" unless you plan to keep this doc updated
          • "not based on the allocated" is missing an object of the preposition
          Show
          templedf Daniel Templeton added a comment - Overall a well written doc. Minor comments: "list loaded nodes" should be "least loaded nodes" "In case of" should be "In the case of" "the NM on a node whose queue length is above the threshold, discards opportunistic containers to meet this maximal value" should drop the comma and be "an NM ..." It would be good to call out the defaults for the properties It's not obvious what the incremental properties do. Min and max are self-explanatory, but I think you have to explain the increments. yarn.opportunistic-container-allocation.nodes-used needs more explanation. I can guess what it does, but it would be better if you just explain it more. yarn.nm-container-queuing.min-queue-length needs more explanation. I have no idea what it does from the doc. How can a minimum be enforced? What if there are no jobs? Same for yarn.nm-container-queuing.min-queue-wait-time-ms "if a map task fail" should be "if a map task fails" "Also, when clicking" should drop the "Also" "the open JIRAs" should just be "the JIRAs" unless you plan to keep this doc updated "not based on the allocated" is missing an object of the preposition
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Thanks for the detailed feedback, Daniel Templeton!
          I also got some offline feedback from Carlo Curino yesterday.

          I will incorporate your changes and upload a new version.

          Regarding the min queue length and wait time, I will improve the description – it is indeed not easy to understand what it does in its current form. These parameters are used to "not dequeue containers for load rebalancing purposes, if queue length is smaller than X tasks (or seconds)". So if you have shorter queues than that, you simply don't perform any action.

          As per Carlo's suggestion too, I will raise a JIRA to simplify some of the properties related to opportunistic containers, including the incremental one. For instance, I don't think there will be many cases where we will want the min/max opportunistic container size to be different from the guaranteed one.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Thanks for the detailed feedback, Daniel Templeton ! I also got some offline feedback from Carlo Curino yesterday. I will incorporate your changes and upload a new version. Regarding the min queue length and wait time, I will improve the description – it is indeed not easy to understand what it does in its current form. These parameters are used to "not dequeue containers for load rebalancing purposes, if queue length is smaller than X tasks (or seconds)". So if you have shorter queues than that, you simply don't perform any action. As per Carlo's suggestion too, I will raise a JIRA to simplify some of the properties related to opportunistic containers, including the incremental one. For instance, I don't think there will be many cases where we will want the min/max opportunistic container size to be different from the guaranteed one.
          Hide
          kasha Karthik Kambatla added a comment -

          Really well written docs, Konstantinos Karanasos.

          High-level comments:

          1. We should honor some character limit for the lines. 80 seems okay, but I am comfortable with longer lines if that makes more sense.
          2. Can we discuss execution before allocation, so we address any skepticism that users have before discussing how the allocation works?
          3. Can we automatically turn on AMRMProxy if someone enables distributed-opportunistic-scheduling?
          4. Rebalancing node load: Since we do not provide details on how the RM arrives at targeted maximal queue length, may be we should keep it simple and just say the RM does it?
          5. The configs don't seem to follow general YARN conventions of yarn.resourcemanager, yarn.nodemanager, or yarn.scheduler. Following these would make it clear what config files each config should make it to. Also, are all the configs necessary? e.g. is clusterwide_min_queue_length necessary? Can't a node not support opportunistic execution? Also, can we not leave the queue lengths to nodes?
          6. In the future work, you mention YARN-5292 (pausing containers), but the description only mentions preemption and never pausing containers.

          I wasn't sure about the best way for minor comments, so will send you a diff capturing my suggested changes offline.

          Show
          kasha Karthik Kambatla added a comment - Really well written docs, Konstantinos Karanasos . High-level comments: We should honor some character limit for the lines. 80 seems okay, but I am comfortable with longer lines if that makes more sense. Can we discuss execution before allocation, so we address any skepticism that users have before discussing how the allocation works? Can we automatically turn on AMRMProxy if someone enables distributed-opportunistic-scheduling? Rebalancing node load: Since we do not provide details on how the RM arrives at targeted maximal queue length, may be we should keep it simple and just say the RM does it? The configs don't seem to follow general YARN conventions of yarn.resourcemanager , yarn.nodemanager , or yarn.scheduler . Following these would make it clear what config files each config should make it to. Also, are all the configs necessary? e.g. is clusterwide_min_queue_length necessary? Can't a node not support opportunistic execution? Also, can we not leave the queue lengths to nodes? In the future work, you mention YARN-5292 (pausing containers), but the description only mentions preemption and never pausing containers. I wasn't sure about the best way for minor comments, so will send you a diff capturing my suggested changes offline.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Thanks for the feedback, Karthik Kambatla!

          Regarding some of your comments:

          We should honor some character limit for the lines.

          The problem is that markdown adds a new line in HTML for each new line in the text, so the HTML ends up not looking good. That's why there are no line wraps in any of the YARN md files I checked.

          Can we discuss execution before allocation?

          I had it like that in an initial version , and then changed it back to follow the actual order of actions (allocation before execution). But I will check it again and might switch it back if it looks better (or add a forward pointer if I keep the current order).

          Can we automatically turn on AMRMProxy if someone enables distributed-opportunistic-scheduling?

          Makes sense, will do.

          The configs don't seem to follow general YARN conventions.

          You are right. I think we had it like that at some point, but things changed across iterations. Will make sure we follow the conventions.

          I got the diff, will check it as well. I will work on the document today or tomorrow and will upload a new version addressing the comments of all of you.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Thanks for the feedback, Karthik Kambatla ! Regarding some of your comments: We should honor some character limit for the lines. The problem is that markdown adds a new line in HTML for each new line in the text, so the HTML ends up not looking good. That's why there are no line wraps in any of the YARN md files I checked. Can we discuss execution before allocation? I had it like that in an initial version , and then changed it back to follow the actual order of actions (allocation before execution). But I will check it again and might switch it back if it looks better (or add a forward pointer if I keep the current order). Can we automatically turn on AMRMProxy if someone enables distributed-opportunistic-scheduling? Makes sense, will do. The configs don't seem to follow general YARN conventions. You are right. I think we had it like that at some point, but things changed across iterations. Will make sure we follow the conventions. I got the diff, will check it as well. I will work on the document today or tomorrow and will upload a new version addressing the comments of all of you.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Hi Andrew,

          Can you wait until the end of the day for this JIRA?
          I am almost done with it and it is the only thing missing for adding the
          opportunistic containers feature to alpha-2.

          Thanks,
          Konstantinos

          Show
          kkaranasos Konstantinos Karanasos added a comment - Hi Andrew, Can you wait until the end of the day for this JIRA? I am almost done with it and it is the only thing missing for adding the opportunistic containers feature to alpha-2. Thanks, Konstantinos
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Adding updated documentation.
          Thanks again for all the feedback, Daniel Templeton, Carlo Curino, Karthik Kambatla, Arun Suresh.
          I have addressed almost all your comments. I also reorganized some sections, adding an initial "Quick Guide" that can be used to quickly enable and tried out the new feature (without requiring to go through all the system details).

          Show
          kkaranasos Konstantinos Karanasos added a comment - Adding updated documentation. Thanks again for all the feedback, Daniel Templeton , Carlo Curino , Karthik Kambatla , Arun Suresh . I have addressed almost all your comments. I also reorganized some sections, adding an initial "Quick Guide" that can be used to quickly enable and tried out the new feature (without requiring to go through all the system details).
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Also updating yarn-default.xml with new variable names.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Also updating yarn-default.xml with new variable names.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 54s Maven dependency ordering for branch
          +1 mvninstall 12m 57s trunk passed
          +1 compile 9m 42s trunk passed
          +1 checkstyle 1m 46s trunk passed
          +1 mvnsite 5m 8s trunk passed
          +1 mvneclipse 3m 26s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 23s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 4m 7s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          +1 mvninstall 4m 0s the patch passed
          +1 compile 9m 25s the patch passed
          +1 javac 9m 25s the patch passed
          -0 checkstyle 1m 49s root: The patch generated 2 new + 954 unchanged - 2 fixed = 956 total (was 956)
          +1 mvnsite 5m 40s the patch passed
          +1 mvneclipse 3m 58s the patch passed
          -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 25s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
          +1 javadoc 4m 47s the patch passed
          +1 unit 0m 37s hadoop-yarn-api in the patch passed.
          +1 unit 2m 30s hadoop-yarn-common in the patch passed.
          +1 unit 13m 48s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 42m 43s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 unit 6m 44s hadoop-yarn-server-tests in the patch failed.
          +1 unit 16m 19s hadoop-yarn-client in the patch passed.
          +1 unit 0m 20s hadoop-yarn-site in the patch passed.
          +1 unit 2m 59s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 8m 50s hadoop-mapreduce-client-app in the patch passed.
          +1 unit 105m 25s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 48s The patch does not generate ASF License warnings.
          312m 52s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.TestContainerManagerSecurity
            hadoop.yarn.server.TestMiniYarnClusterNodeUtilization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5646
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843539/YARN-5646.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e595232a0381 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cee0c46
          Default Java 1.8.0_111
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14342/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14342/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 54s Maven dependency ordering for branch +1 mvninstall 12m 57s trunk passed +1 compile 9m 42s trunk passed +1 checkstyle 1m 46s trunk passed +1 mvnsite 5m 8s trunk passed +1 mvneclipse 3m 26s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 23s branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 4m 7s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch +1 mvninstall 4m 0s the patch passed +1 compile 9m 25s the patch passed +1 javac 9m 25s the patch passed -0 checkstyle 1m 49s root: The patch generated 2 new + 954 unchanged - 2 fixed = 956 total (was 956) +1 mvnsite 5m 40s the patch passed +1 mvneclipse 3m 58s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 25s patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests no findbugs output file (hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml) +1 javadoc 4m 47s the patch passed +1 unit 0m 37s hadoop-yarn-api in the patch passed. +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 13m 48s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 42m 43s hadoop-yarn-server-resourcemanager in the patch failed. -1 unit 6m 44s hadoop-yarn-server-tests in the patch failed. +1 unit 16m 19s hadoop-yarn-client in the patch passed. +1 unit 0m 20s hadoop-yarn-site in the patch passed. +1 unit 2m 59s hadoop-mapreduce-client-core in the patch passed. +1 unit 8m 50s hadoop-mapreduce-client-app in the patch passed. +1 unit 105m 25s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 48s The patch does not generate ASF License warnings. 312m 52s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.TestContainerManagerSecurity   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5646 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843539/YARN-5646.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e595232a0381 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cee0c46 Default Java 1.8.0_111 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14342/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14342/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/14342/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          asuresh Arun Suresh added a comment -

          The checkstyle warnings can be ignored to retain the style of existing variables in MRJobConfig
          The findbugs warnings are not related and appear because I think mvn could not find the findBugs.xml file.
          The test failures are also unrelated and seem to work fine locally (except for TestContainerManagerSecurity for which, i think we need to re-open YARN-5655)

          Committing this to trunk shortly.

          Show
          asuresh Arun Suresh added a comment - The checkstyle warnings can be ignored to retain the style of existing variables in MRJobConfig The findbugs warnings are not related and appear because I think mvn could not find the findBugs.xml file. The test failures are also unrelated and seem to work fine locally (except for TestContainerManagerSecurity for which, i think we need to re-open YARN-5655 ) Committing this to trunk shortly.
          Hide
          asuresh Arun Suresh added a comment -

          Committed this to trunk. Thanks for the patch Konstantinos Karanasos and for the reviews Karthik Kambatla, Daniel Templeton and Carlo Curino

          Show
          asuresh Arun Suresh added a comment - Committed this to trunk. Thanks for the patch Konstantinos Karanasos and for the reviews Karthik Kambatla , Daniel Templeton and Carlo Curino
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11007 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11007/)
          YARN-5646. Add documentation and update config parameter names for (arun suresh: rev 2273a74c1f3895163046cca09ff5e983df301d22)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/OpportunisticContainers.md
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMROpportunisticMaps.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestDistributedScheduling.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11007 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11007/ ) YARN-5646 . Add documentation and update config parameter names for (arun suresh: rev 2273a74c1f3895163046cca09ff5e983df301d22) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/OpportunisticContainers.md (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMROpportunisticMaps.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestDistributedScheduling.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Thanks Arun Suresh!

          Show
          kkaranasos Konstantinos Karanasos added a comment - Thanks Arun Suresh !
          Hide
          asuresh Arun Suresh added a comment -

          Committing this to branch-2

          Show
          asuresh Arun Suresh added a comment - Committing this to branch-2
          Hide
          asuresh Arun Suresh added a comment -

          Reopening to add addendum patch to include changes to the site.xml file

          Show
          asuresh Arun Suresh added a comment - Reopening to add addendum patch to include changes to the site.xml file
          Hide
          asuresh Arun Suresh added a comment -

          Committed addendum patch to trunk and branch-2

          Show
          asuresh Arun Suresh added a comment - Committed addendum patch to trunk and branch-2

            People

            • Assignee:
              kkaranasos Konstantinos Karanasos
              Reporter:
              kkaranasos Konstantinos Karanasos
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development