[YARN-9921] Issue in PlacementConstraint when YARN Service AM retries allocation on component failure.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.3.0, 3.1.4
• Assignee: Tarun Parimi
• Reporter: Tarun Parimi
• Created: 21/Oct/19 05:05
• Updated: 24/Oct/19 07:19
• Resolved: 24/Oct/19 02:17

Description

When YARN Service AM tries to relaunch a container on failure, we encounter the below error in PlacementConstraints.

ERROR impl.AMRMClientAsyncImpl: Exception on heartbeat
org.apache.hadoop.yarn.exceptions.YarnException: org.apache.hadoop.yarn.exceptions.SchedulerInvalidResoureRequestException: Invalid updated SchedulingRequest added to scheduler, we only allows changing numAllocations for the updated SchedulingRequest. Old=SchedulingRequestPBImpl{priority=0, allocationReqId=0, executionType={Execution Type: GUARANTEED, Enforce Execution Type: true}, allocationTags=[component], resourceSizing=ResourceSizingPBImpl{numAllocations=0, resources=<memory:557056, vCores:1>}, placementConstraint=notin,node,llap:notin,node,yarn_node_partition/=[label]} new=SchedulingRequestPBImpl{priority=0, allocationReqId=0, executionType={Execution Type: GUARANTEED, Enforce Execution Type: true}, allocationTags=[component], resourceSizing=ResourceSizingPBImpl{numAllocations=1, resources=<memory:557056, vCores:1>}, placementConstraint=notin,node,component:notin,node,yarn_node_partition/=[label]}, if any fields need to be updated, please cancel the old request (by setting numAllocations to 0) and send a SchedulingRequest with different combination of priority/allocationId

But we can see from the message that the SchedulingRequest is indeed valid with everything same except numAllocations as expected. But still the below equals check in SingleConstraintAppPlacementAllocator fails.

// Compare two objects
      if (!schedulingRequest.equals(newSchedulingRequest)) {
        // Rollback #numAllocations
        sizing.setNumAllocations(newNumAllocations);
        throw new SchedulerInvalidResoureRequestException(
            "Invalid updated SchedulingRequest added to scheduler, "
                + " we only allows changing numAllocations for the updated "
                + "SchedulingRequest. Old=" + schedulingRequest.toString()
                + " new=" + newSchedulingRequest.toString()
                + ", if any fields need to be updated, please cancel the "
                + "old request (by setting numAllocations to 0) and send a "
                + "SchedulingRequest with different combination of "
                + "priority/allocationId");
      }