Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3823

Fix mismatch in default values for yarn.scheduler.maximum-allocation-vcores property

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Default value for 'yarn.scheduler.maximum-allocation-vcores' changed from 32 to 4.

      Description

      In yarn-default.xml, the property is defined as:

      XML Property: yarn.scheduler.maximum-allocation-vcores
      XML Value: 32

      In YarnConfiguration.java the corresponding member variable is defined as:

      Config Name: DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES
      Config Value: 4

      The Config value comes from YARN-193 and the default xml property comes from YARN-2. Should we keep it this way or should one of the values get updated?

      1. YARN-3823.001.patch
        1 kB
        Ray Chiang
      2. YARN-3823.002.patch
        0.7 kB
        Ray Chiang

        Activity

        Hide
        rchiang Ray Chiang added a comment -

        Initial version. Assume YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES needs to be increased from 4 to 32.

        Show
        rchiang Ray Chiang added a comment - Initial version. Assume YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES needs to be increased from 4 to 32.
        Hide
        rchiang Ray Chiang added a comment -

        Submit for testing

        Show
        rchiang Ray Chiang added a comment - Submit for testing
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 45s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 8m 7s There were no new javac warning messages.
        +1 javadoc 11m 50s There were no new javadoc warning messages.
        +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 1m 15s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 2m 9s mvn install still works.
        +1 eclipse:eclipse 0m 41s The patch built with eclipse:eclipse.
        +1 findbugs 1m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
            43m 29s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742621/YARN-3823.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 8e33372
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8380/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8380/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8380/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 8m 7s There were no new javac warning messages. +1 javadoc 11m 50s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 15s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 2m 9s mvn install still works. +1 eclipse:eclipse 0m 41s The patch built with eclipse:eclipse. +1 findbugs 1m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.     43m 29s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742621/YARN-3823.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8e33372 hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8380/artifact/patchprocess/testrun_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8380/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8380/console This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        RE: No new tests

        The unit test is in a different JIRA. HADOOP-12101 actually detects this mismatch.

        Show
        rchiang Ray Chiang added a comment - RE: No new tests The unit test is in a different JIRA. HADOOP-12101 actually detects this mismatch.
        Hide
        devaraj.k Devaraj K added a comment -

        Ray Chiang, Your change makes sense to me.

        YARN-193 added this config with the default value as 32 in both of the places, Did you find any jira which changed value from 32 to 4 in YarnConfiguration.java?

        Show
        devaraj.k Devaraj K added a comment - Ray Chiang , Your change makes sense to me. YARN-193 added this config with the default value as 32 in both of the places, Did you find any jira which changed value from 32 to 4 in YarnConfiguration.java?
        Hide
        rchiang Ray Chiang added a comment -

        I see this for YARN-193, using "git blame bc6777dd5bdcbaef09897b506bc6511ae456033d"

        • public static final String RM_SCHEDULER_MAXIMUM_ALLOCATION_CORES =
          + public static final String RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES =
          YARN_PREFIX + "scheduler.maximum-allocation-vcores";
        • public static final int DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_CORES = 32;
          + public static final int DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES = 4;

        which looks like moving from 32 to 4 in YarnConfiguration.java while renaming _CORES to _VCORES.

        I'm not sure if the intended semantics changed during this property rename. Zhijie Shen, Vinod Kumar Vavilapalli do either of you have any comment on this?

        Show
        rchiang Ray Chiang added a comment - I see this for YARN-193 , using "git blame bc6777dd5bdcbaef09897b506bc6511ae456033d" public static final String RM_SCHEDULER_MAXIMUM_ALLOCATION_CORES = + public static final String RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES = YARN_PREFIX + "scheduler.maximum-allocation-vcores"; public static final int DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_CORES = 32; + public static final int DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES = 4; which looks like moving from 32 to 4 in YarnConfiguration.java while renaming _CORES to _VCORES. I'm not sure if the intended semantics changed during this property rename. Zhijie Shen , Vinod Kumar Vavilapalli do either of you have any comment on this?
        Hide
        zjshen Zhijie Shen added a comment -

        Hm... I think we forgot changing the default in the config file from 32 to 4. Compared to 8GB default max mem, 32 vcores are too big to be a reasonable machine. See the comment:

        https://issues.apache.org/jira/browse/YARN-193?focusedCommentId=13621073&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621073

        Show
        zjshen Zhijie Shen added a comment - Hm... I think we forgot changing the default in the config file from 32 to 4. Compared to 8GB default max mem, 32 vcores are too big to be a reasonable machine. See the comment: https://issues.apache.org/jira/browse/YARN-193?focusedCommentId=13621073&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13621073
        Hide
        rchiang Ray Chiang added a comment -

        That makes sense. I'll update yarn-default.xml to match YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES then. Thanks!

        Show
        rchiang Ray Chiang added a comment - That makes sense. I'll update yarn-default.xml to match YarnConfiguration.DEFAULT_RM_SCHEDULER_MAXIMUM_ALLOCATION_VCORES then. Thanks!
        Hide
        rchiang Ray Chiang added a comment -
        • Adjust yarn.scheduler.maximum-allocation-vcores to 4 in yarn-default.xml to match YarnConfiguration
        Show
        rchiang Ray Chiang added a comment - Adjust yarn.scheduler.maximum-allocation-vcores to 4 in yarn-default.xml to match YarnConfiguration
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 58s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 38s There were no new javac warning messages.
        +1 javadoc 9m 37s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
            36m 41s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12742911/YARN-3823.002.patch
        Optional Tests javadoc javac unit
        git revision trunk / 147e020
        hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8394/artifact/patchprocess/testrun_hadoop-yarn-common.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8394/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8394/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 38s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.     36m 41s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742911/YARN-3823.002.patch Optional Tests javadoc javac unit git revision trunk / 147e020 hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8394/artifact/patchprocess/testrun_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8394/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8394/console This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        RE: No new unit test

        Same as before. The unit test is in a different JIRA. HADOOP-12101 actually detects this mismatch.

        Show
        rchiang Ray Chiang added a comment - RE: No new unit test Same as before. The unit test is in a different JIRA. HADOOP-12101 actually detects this mismatch.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Zhijie Shen for confirmation.

        +1 for the latest patch.

        Show
        devaraj.k Devaraj K added a comment - Thanks Zhijie Shen for confirmation. +1 for the latest patch.
        Hide
        devaraj.k Devaraj K added a comment -

        Thanks Ray Chiang for the patch.

        Committed to trunk and branch-2.

        Show
        devaraj.k Devaraj K added a comment - Thanks Ray Chiang for the patch. Committed to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #8104 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8104/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8104 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8104/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #975 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/975/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #245 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/245/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2172 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2172/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #233 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/233/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2191/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/)
        YARN-3823. Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #243 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/243/ ) YARN-3823 . Fix mismatch in default values for (devaraj: rev 7405c59799ed1b8ad1a7c6f1b18fabf49d0b92b2) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
        Hide
        rchiang Ray Chiang added a comment -

        Double thanks for the info.

        Show
        rchiang Ray Chiang added a comment - Double thanks for the info.
        Hide
        rchiang Ray Chiang added a comment -

        Thanks for the review and commit!

        Show
        rchiang Ray Chiang added a comment - Thanks for the review and commit!

          People

          • Assignee:
            rchiang Ray Chiang
            Reporter:
            rchiang Ray Chiang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development