Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4256

YARN fair scheduler vcores with decimal values

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.7.1
    • 2.8.0, 3.0.0-alpha1
    • fairscheduler
    • None
    • Reviewed

    Description

      When the queue with vcores is in decimal value, the value after the decimal point is taken as vcores by FairScheduler.

      For the below queue,

      <minResources>20000 mb,20 vcores,20.25 disks</minResources>
      <maxResources>30000 mb,40.2 vcores,30.25 disks</maxResources>

      When many applications submitted parallely into queue, all were in PENDING state as the vcores is taken as 2 skipping the value 40.

      The code FairSchedulerConfiguration.java to Pattern match the vcores has to be improved in such a way either throw AllocationConfigurationException("Missing resource") or consider the value before decimal.

      Attachments

        1. YARN-4256.001.patch
          3 kB
          Jun Gong
        2. YARN-4256.002.patch
          3 kB
          Jun Gong

        Activity

          hex108 Jun Gong added a comment - - edited

          Thanks prabhujoseph for reporting. How about considering the value before decimal? If OK, I will attach a patch.

          hex108 Jun Gong added a comment - - edited Thanks prabhujoseph for reporting. How about considering the value before decimal? If OK, I will attach a patch.
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 48s There were no new javac warning messages.
          +1 javadoc 11m 47s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 55s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 41s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 63m 3s Tests passed in hadoop-yarn-server-resourcemanager.
              108m 4s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12766517/YARN-4256.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d6c8bad
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9440/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9440/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9440/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 48s There were no new javac warning messages. +1 javadoc 11m 47s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 55s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 41s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 63m 3s Tests passed in hadoop-yarn-server-resourcemanager.     108m 4s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12766517/YARN-4256.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d6c8bad hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9440/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9440/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9440/console This message was automatically generated.
          prabhujoseph Prabhu Joseph added a comment -

          Thanks Jun Gong.

          prabhujoseph Prabhu Joseph added a comment - Thanks Jun Gong.
          hex108 Jun Gong added a comment -

          zxu, could you please help review it?

          hex108 Jun Gong added a comment - zxu , could you please help review it?
          zxu Zhihai Xu added a comment - - edited

          Thanks for reporting this issue prabhujoseph! Thanks for the patch hex108! The patch looks most good. Can we change '+' to '*'
          (\\.
          d+)? => (\\.
          d*)? So we can relax the condition to support 1024. mb.

          zxu Zhihai Xu added a comment - - edited Thanks for reporting this issue prabhujoseph ! Thanks for the patch hex108 ! The patch looks most good. Can we change '+' to '*' (\\. d+)? => (\\. d*)? So we can relax the condition to support 1024. mb.
          hex108 Jun Gong added a comment -

          Thanks zxu for the review and comments.

          Attach a new patch to address it and add a new test for the case '1024. mb'.

          hex108 Jun Gong added a comment - Thanks zxu for the review and comments. Attach a new patch to address it and add a new test for the case '1024. mb'.
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 9m 32s There were no new javac warning messages.
          +1 javadoc 11m 47s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 57s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 48s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 1m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 64m 35s Tests passed in hadoop-yarn-server-resourcemanager.
              110m 49s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767706/YARN-4256.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0c4af0f
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9504/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9504/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9504/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 32s There were no new javac warning messages. +1 javadoc 11m 47s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 57s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 48s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 1m 36s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 64m 35s Tests passed in hadoop-yarn-server-resourcemanager.     110m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767706/YARN-4256.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0c4af0f hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9504/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9504/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9504/console This message was automatically generated.
          zxu Zhihai Xu added a comment -

          +1 LGTM, Will commit tomorrow if no one objects.

          zxu Zhihai Xu added a comment - +1 LGTM, Will commit tomorrow if no one objects.
          brahmareddy Brahma Reddy Battula added a comment - - edited

          prabhujoseph nice finding..hex108 thanks for patch..LGTM..

          brahmareddy Brahma Reddy Battula added a comment - - edited prabhujoseph nice finding.. hex108 thanks for patch..LGTM..
          zxu Zhihai Xu added a comment - - edited

          committed it to trunk and branch-2, Thanks prabhujoseph for reporting this issue, thanks hex108 for the patch and thanks brahmareddy for additional review!

          zxu Zhihai Xu added a comment - - edited committed it to trunk and branch-2, Thanks prabhujoseph for reporting this issue, thanks hex108 for the patch and thanks brahmareddy for additional review!
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #570 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/570/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #570 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/570/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8691/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8691 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8691/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1306 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1306/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1306 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1306/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #585 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/585/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #585 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/585/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2517/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2517 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2517/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hex108 Jun Gong added a comment -

          Thanks zxu for the review and commit!

          hex108 Jun Gong added a comment - Thanks zxu for the review and commit!
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2465 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2465/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2465 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2465/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #528 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/528/)
          YARN-4256. YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #528 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/528/ ) YARN-4256 . YARN fair scheduler vcores with decimal values. Contributed (zxu: rev 960201b79b9f2ca40f8eadb21a2f9fe37dde2b5d) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java

          People

            hex108 Jun Gong
            prabhujoseph Prabhu Joseph
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: