Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-11014

YARN incorrectly validates maximum capacity resources on the validation API

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      The YARN validation API (http://<host>:8088/ws/v1/cluster/scheduler-conf/validate) allows a configuration in which a child queue has a greater maximum capacity value than its parent (the same update fails when it is attempted for real) using the DominantResourceCalculator.
      For example, the following passes validation when the root queue's maximum capacity is less than the 100000000 it is attempting to set here:

      {
          "update-queue": [
              {
                  "queue-name": "root.test",
                  "params": {
                      "entry": [
                          {
                              "key": "maximum-capacity",
                              "value": "[memory=100000000,vcores=3]"
                          },
                          {
                              "key": "capacity",
                              "value": "[memory=3020,vcores=1]"
                          }
                      ]
                  }
              }
          ]
      } 
      

      The reason for this is the newly created CapacityScheduler instance doesn't have the active nodes of the cluster in the nodeTracker object, hence the clusterResources will be Resources.none() during the init/reinit phase of this new instance.
      If the dominantRC has invalid denominator (clusterResource being 0) it simply compares the two values resource-by-resource:

        @Override
        public int compare(Resource clusterResource, Resource lhs, Resource rhs,
            boolean singleType) {
          if (lhs.equals(rhs)) {
            return 0;
          }
      
          if (isAllInvalidDivisor(clusterResource)) {
            return this.compare(lhs, rhs);
          }
      ...
      
        /**
         * Compare two resources - if the value for every resource type for the lhs
         * is greater than that of the rhs, return 1. If the value for every resource
         * type in the lhs is less than the rhs, return -1. Otherwise, return 0
         *
         * @param lhs resource to be compared
         * @param rhs resource to be compared
         * @return 0, 1, or -1
         */
         private int compare(Resource lhs, Resource rhs) {
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bteke Benjamin Teke Assign to me
            bteke Benjamin Teke
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 3h 10m
              3h 10m

              Slack

                Issue deployment