[YARN-8248] Job hangs when a job requests a resource that its queue does not have

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: fairscheduler
• Target Version/s: 3.2.0, 3.1.1
• Hadoop Flags: Reviewed
• Assignee: Szilard Nemeth
• Reporter: Szilard Nemeth
• Created: 04/May/18 13:06
• Updated: 03/Feb/19 05:25
• Resolved: 21/May/18 15:10

Description

Job hangs when mapreduce.job.queuename is specified and the queue has 0 of any resource (vcores / memory / other)

In this scenario, the job should be immediately rejected upon submission since the specified queue cannot serve the resource needs of the submitted job.

 

Command to run:

bin/yarn jar "./share/hadoop/mapreduce/hadoop-mapreduce-examples-$MY_HADOOP_VERSION.jar" pi -Dmapreduce.job.queuename=sample_queue 1 1000;

fair-scheduler.xml queue config (excerpt):

 

 <queue name="sample_queue">
    <minResources>10000 mb,0vcores</minResources>
    <maxResources>90000 mb,0vcores</maxResources>
    <maxRunningApps>50</maxRunningApps>
    <maxAMShare>-1.0f</maxAMShare>
    <weight>2.0</weight>
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>

Diagnostic message from the web UI: 

Wed May 02 06:35:57 -0700 2018] Application is added to the scheduler and is not yet activated. (Resource request: <memory:1536, vCores:1> exceeds current queue or its parents maximum resource allowed).

Issue Links

• is related to: YARN-9276 Filter non-existent resource requests actively