Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4922

Request with multiple data local nodes can cause NPE in AppSchedulingInfo

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: None
    • Labels:
      None

      Description

      With the way that the schedulers work, each request for a container on a node must consist of 3 ResourceRequests - one on the node, one on the rack, and one with *.

      AppSchedulingInfo tracks the outstanding requests. When a node is assigned a node-local container, allocateNodeLocal decrements the outstanding requests at each level - node, rack, and *. If the rack requests reach 0, it removes the mapping.

      A mapreduce task with multiple data local nodes submits multiple container requests, one for each node. It also submits one for each unique rack, and one for *. If there are fewer unique racks than data local nodes, this means that fewer rack-local ResourceRequests will be submitted than node-local ResourceRequests, so the rack-local mapping will be deleted before all the node-local requests are allocated and an NPE will come up the next time a node-local request from that rack is allocated.

        Attachments

          Activity

            People

            • Assignee:
              sandyr Sandy Ryza
              Reporter:
              sandyr Sandy Ryza
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: