Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4256

YARN fair scheduler vcores with decimal values

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When the queue with vcores is in decimal value, the value after the decimal point is taken as vcores by FairScheduler.

      For the below queue,

      <minResources>20000 mb,20 vcores,20.25 disks</minResources>
      <maxResources>30000 mb,40.2 vcores,30.25 disks</maxResources>

      When many applications submitted parallely into queue, all were in PENDING state as the vcores is taken as 2 skipping the value 40.

      The code FairSchedulerConfiguration.java to Pattern match the vcores has to be improved in such a way either throw AllocationConfigurationException("Missing resource") or consider the value before decimal.

      1. YARN-4256.001.patch
        3 kB
        Jun Gong
      2. YARN-4256.002.patch
        3 kB
        Jun Gong

        Activity

        Hide
        hex108 Jun Gong added a comment -

        Thanks Prabhu Joseph for reporting. How about considering the value before decimal? If OK, I will attach a patch.

        Show
        hex108 Jun Gong added a comment - Thanks Prabhu Joseph for reporting. How about considering the value before decimal? If OK, I will attach a patch.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 4s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 8m 48s There were no new javac warning messages.
        +1 javadoc 11m 47s There were no new javadoc warning messages.
        +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 55s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 41s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 63m 3s Tests passed in hadoop-yarn-server-resourcemanager.
            108m 4s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12766517/YARN-4256.001.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d6c8bad
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9440/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9440/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9440/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 48s There were no new javac warning messages. +1 javadoc 11m 47s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 55s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 63m 3s Tests passed in hadoop-yarn-server-resourcemanager.     108m 4s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766517/YARN-4256.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d6c8bad hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9440/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9440/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9440/console This message was automatically generated.
        Hide
        Prabhu Joseph Prabhu Joseph added a comment -

        Thanks Jun Gong.

        Show
        Prabhu Joseph Prabhu Joseph added a comment - Thanks Jun Gong.
        Hide
        hex108 Jun Gong added a comment -

        zhihai xu, could you please help review it?

        Show
        hex108 Jun Gong added a comment - zhihai xu , could you please help review it?
        Hide
        zxu zhihai xu added a comment -

        Thanks for reporting this issue Prabhu Joseph! Thanks for the patch Jun Gong! The patch looks most good. Can we change '+' to '*'
        (\\.
        d+)? => (\\.
        d*)? So we can relax the condition to support 1024. mb.

        Show
        zxu zhihai xu added a comment - Thanks for reporting this issue Prabhu Joseph ! Thanks for the patch Jun Gong ! The patch looks most good. Can we change '+' to '*' (\\. d+)? => (\\. d*)? So we can relax the condition to support 1024. mb.
        Hide
        hex108 Jun Gong added a comment -

        Thanks zhihai xu for the review and comments.

        Attach a new patch to address it and add a new test for the case '1024. mb'.

        Show
        hex108 Jun Gong added a comment - Thanks zhihai xu for the review and comments. Attach a new patch to address it and add a new test for the case '1024. mb'.
        Hide
        hadoopqa Hadoop QA added a comment -



        +1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 19m 26s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 32s There were no new javac warning messages.
        +1 javadoc 11m 47s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 0m 57s There were no new checkstyle issues.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 48s mvn install still works.
        +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
        +1 findbugs 1m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 64m 35s Tests passed in hadoop-yarn-server-resourcemanager.
            110m 49s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12767706/YARN-4256.002.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 0c4af0f
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9504/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9504/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9504/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 32s There were no new javac warning messages. +1 javadoc 11m 47s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 57s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 48s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 1m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 64m 35s Tests passed in hadoop-yarn-server-resourcemanager.     110m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767706/YARN-4256.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0c4af0f hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9504/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9504/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9504/console This message was automatically generated.
        Hide
        zxu zhihai xu added a comment -

        +1 LGTM, Will commit tomorrow if no one objects.

        Show
        zxu zhihai xu added a comment - +1 LGTM, Will commit tomorrow if no one objects.
        Hide
        brahmareddy Brahma Reddy Battula added a comment -

        Prabhu Joseph nice finding..Jun Gong thanks for patch..LGTM..

        Show
        brahmareddy Brahma Reddy Battula added a comment - Prabhu Joseph nice finding.. Jun Gong thanks for patch..LGTM..
        Hide
        zxu zhihai xu added a comment -

        committed it to trunk and branch-2, Thanks Prabhu Joseph for reporting this issue, thanks Jun Gong for the patch and thanks Brahma Reddy Battula for additional review!

        Show
        zxu zhihai xu added a comment - committed it to trunk and branch-2, Thanks Prabhu Joseph for reporting this issue, thanks Jun Gong for the patch and thanks Brahma Reddy Battula for additional review!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #570 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/570/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #570 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/570/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8691/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8691/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #1306 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1306/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1306 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1306/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #585 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/585/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #585 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/585/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2517/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2517/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Hide
        hex108 Jun Gong added a comment -

        Thanks zhihai xu for the review and commit!

        Show
        hex108 Jun Gong added a comment - Thanks zhihai xu for the review and commit!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2465 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2465/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2465 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2465/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #528 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/528/)
        YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #528 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/528/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java

          People

          • Assignee:
            hex108 Jun Gong
            Reporter:
            Prabhu Joseph Prabhu Joseph
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development