Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10705

Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4.0
    • Fix Version/s: 3.4.0
    • Component/s: yarn
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Following DEBUG logs are logged if a container reservation is made when a node has been offered to the queue in FairScheduler:

      2021-02-10 07:33:55,049 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: application_1610442362681_2607's resource request is reserved.
      2021-02-10 07:33:55,049 DEBUG org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue: Assigned container in queue:root.pj_dc_pe container:<memory:-1, vCores:0>
      

      The latter log from above seems to indicate a bad container assignment with <memory:-1, vCores:0> resource allocation, whereas, in actual, it is a bad log which shouldn't have been logged in the first place.

      This log comes from [1] after an application attempt with an unmet demand is checked for container assignment/reservation.

      If the container for this app attempt is reserved on the node, then, it returns <memory:-1, vCores:0> from [2].

      From [3]:

      • If an assignment was made, returns the resources allocated to the
      • container. If a reservation was made, returns
      • FairScheduler.CONTAINER_RESERVED. If no assignment or reservation was
      • made, returns an empty resource.

      We are checking for the empty resource at [4], but not FairScheduler.CONTAINER_RESERVED before logging out a message for container assignment specifically which is incorrect.

      Instead of:

            if (!assigned.equals(none())) {
              LOG.debug("Assigned container in queue:{} container:{}",
                  getName(), assigned);
              break;
            }
      

      it should be:

            // check if an assignment or a reservation was made.
            if (!assigned.equals(none())) {
              // only log container assignment if there is
              // an actual assignment, not a reservation.
              if (!assigned.equals(FairScheduler.CONTAINER_RESERVED)
                      && LOG.isDebugEnabled()) {
                LOG.debug("Assigned container in queue:" + getName() + " " +
                              "container:" + assigned);
              }
              break;
            }
      

      [1] https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L356
      [2] https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L911
      [3] https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L842
      [4] https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L355

        Attachments

        1. YARN-10705.001.patch
          2 kB
          Siddharth Ahuja

          Activity

            People

            • Assignee:
              sahuja Siddharth Ahuja
              Reporter:
              sahuja Siddharth Ahuja
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: