Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6057

yarn.scheduler.minimum-allocation-* descriptions are incorrect about behavior when a request is out of bounds

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

        <property>
          <description>The minimum allocation for every container request at the RM,
          in terms of virtual CPU cores. Requests lower than this will throw a
          InvalidResourceRequestException.</description>
          <name>yarn.scheduler.minimum-allocation-vcores</name>
          <value>1</value>
        </property>
      

      Requests lower than this will throw a InvalidResourceRequestException. Only incase of maximum allocation vcore and memory InvalidResourceRequestException is thrown

      1. YARN-6057.001.patch
        3 kB
        Julia Sommer
      2. YARN-6057.002.patch
        3 kB
        Julia Sommer

        Activity

        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        IIUC for minimum allocation the requests gets roundup to minimum value/ increment resource value

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited IIUC for minimum allocation the requests gets roundup to minimum value/ increment resource value
        Hide
        templedf Daniel Templeton added a comment -

        In the case of both minimum and maximum, resource requests that are out of bounds are quietly adjusted to be in bounds. (See DefaultResourceCalculator.normalize() and DominantResourceCalculator.normalize()) The minimum will also prevent NMs that have fewer vcores from starting. (See ResourceTrackerService.registerNodeManager().)

        Show
        templedf Daniel Templeton added a comment - In the case of both minimum and maximum, resource requests that are out of bounds are quietly adjusted to be in bounds. (See DefaultResourceCalculator.normalize() and DominantResourceCalculator.normalize() ) The minimum will also prevent NMs that have fewer vcores from starting. (See ResourceTrackerService.registerNodeManager() .)
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Daniel Templeton
        In case of resource request greater than maximum InvalidResourceRequest Exception will the thrown but not for minimum.
        Both gets adjusted to be in bounds based on resource calculator.

        Julia Sommer
        Could you provide a patch for the same. I can help in review and commit

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Daniel Templeton In case of resource request greater than maximum InvalidResourceRequest Exception will the thrown but not for minimum. Both gets adjusted to be in bounds based on resource calculator. Julia Sommer Could you provide a patch for the same. I can help in review and commit
        Hide
        Juliasommer Julia Sommer added a comment -

        Changed description to reflect values getting adjusted to match rather than shooting an error code.

        Show
        Juliasommer Julia Sommer added a comment - Changed description to reflect values getting adjusted to match rather than shooting an error code.
        Hide
        templedf Daniel Templeton added a comment -

        LGTM +1 pending Jenkins. Looks like pre-commit is gummed up, so it might be a while.

        Show
        templedf Daniel Templeton added a comment - LGTM +1 pending Jenkins. Looks like pre-commit is gummed up, so it might be a while.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 15m 33s trunk passed
        +1 compile 0m 35s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 javadoc 0m 31s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 28s the patch passed
        +1 javac 0m 28s the patch passed
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 javadoc 0m 30s the patch passed
        +1 unit 2m 36s hadoop-yarn-common in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        23m 47s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6057
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846104/YARN-6057.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 59d94072e986 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 71a4acf
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14595/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14595/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 33s trunk passed +1 compile 0m 35s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 15s trunk passed +1 javadoc 0m 31s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 28s the patch passed +1 javac 0m 28s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 30s the patch passed +1 unit 2m 36s hadoop-yarn-common in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 23m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6057 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846104/YARN-6057.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 59d94072e986 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 71a4acf Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14595/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14595/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Thank you Julia Sommer for updating patch

        For yarn.scheduler.maximum-allocation-* documentation update

        Memory requests higher than this will be set to the value of this property

        -1 from my side

        Validation for resource requests happens first

        Application master container request
        Please check RMAppManager#createAndPopulateNewRMApp -> validateAndCreateResourceRequest

        For ApplicationMasterProtocol container request from AM

        Please check ApplicationMasterService#allocate -> ApplicationMasterService#allocateInternal

        Show
        bibinchundatt Bibin A Chundatt added a comment - Thank you Julia Sommer for updating patch For yarn.scheduler.maximum-allocation-* documentation update Memory requests higher than this will be set to the value of this property -1 from my side Validation for resource requests happens first Application master container request Please check RMAppManager#createAndPopulateNewRMApp -> validateAndCreateResourceRequest For ApplicationMasterProtocol container request from AM Please check ApplicationMasterService#allocate -> ApplicationMasterService#allocateInternal
        Hide
        templedf Daniel Templeton added a comment -

        That's my bad. I foolishly assumed that the max and min would be used symmetrically and didn't track down use of the max value independently. Julia Sommer, can you please drop the changes to the maximum values and post a new patch?

        Show
        templedf Daniel Templeton added a comment - That's my bad. I foolishly assumed that the max and min would be used symmetrically and didn't track down use of the max value independently. Julia Sommer , can you please drop the changes to the maximum values and post a new patch?
        Hide
        Juliasommer Julia Sommer added a comment -

        Corrected version attached. Thanks!

        Show
        Juliasommer Julia Sommer added a comment - Corrected version attached. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 14m 24s trunk passed
        +1 compile 0m 30s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 javadoc 0m 29s trunk passed
        +1 mvninstall 0m 26s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 javadoc 0m 29s the patch passed
        +1 unit 2m 30s hadoop-yarn-common in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        21m 46s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6057
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846407/YARN-6057.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 0ed77c2a909e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 91bf504
        Default Java 1.8.0_111
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14610/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14610/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 14m 24s trunk passed +1 compile 0m 30s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 13s trunk passed +1 javadoc 0m 29s trunk passed +1 mvninstall 0m 26s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 29s the patch passed +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 21m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6057 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12846407/YARN-6057.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 0ed77c2a909e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 91bf504 Default Java 1.8.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14610/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common Console output https://builds.apache.org/job/PreCommit-YARN-Build/14610/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        LGTM +1 . Will commit it in soon.

        Show
        bibinchundatt Bibin A Chundatt added a comment - LGTM +1 . Will commit it in soon.
        Hide
        bibinchundatt Bibin A Chundatt added a comment - - edited

        Committed to trunk and branch-2

        Thank you Julia Sommer for patch and Daniel Templeton for review

        Show
        bibinchundatt Bibin A Chundatt added a comment - - edited Committed to trunk and branch-2 Thank you Julia Sommer for patch and Daniel Templeton for review
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11126 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11126/)
        YARN-6057. yarn.scheduler.minimum-allocation-* descriptions are (bibinchundatt: rev a9d946d4df91c29ea86851bb1c5046e95dc80406)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11126 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11126/ ) YARN-6057 . yarn.scheduler.minimum-allocation-* descriptions are (bibinchundatt: rev a9d946d4df91c29ea86851bb1c5046e95dc80406) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

          People

          • Assignee:
            Juliasommer Julia Sommer
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development