Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-412

FifoScheduler incorrectly checking for node locality

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: scheduler
    • Labels:

      Description

      In the FifoScheduler, the assignNodeLocalContainers method is checking if the data is local to a node by searching for the nodeAddress of the node in the set of outstanding requests for the app. This seems to be incorrect as it should be checking hostname instead. The offending line of code is 455:

      application.getResourceRequest(priority, node.getRMNode().getNodeAddress());

      Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses are a concatenation of hostname and command port (e.g. host1.foo.com:1234)

      In the CapacityScheduler, it's done using hostname. See LeafQueue.assignNodeLocalContainers, line 1129

      application.getResourceRequest(priority, node.getHostName());

      Note that this bug does not affect the actual scheduling decisions made by the FifoScheduler because even though it incorrect determines that a request is not local to the node, it will still schedule the request immediately because it's rack-local. However, this bug may be adversely affecting the reporting of job status by underreporting the number of tasks that were node local.

        Attachments

        1. YARN-412.patch
          10 kB
          Roger Hoover

          Activity

            People

            • Assignee:
              theduderog Roger Hoover
              Reporter:
              theduderog Roger Hoover
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: