Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8248

Job hangs when a job requests a resource that its queue does not have

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Job hangs when mapreduce.job.queuename is specified and the queue has 0 of any resource (vcores / memory / other)

      In this scenario, the job should be immediately rejected upon submission since the specified queue cannot serve the resource needs of the submitted job.

       

      Command to run:

      bin/yarn jar "./share/hadoop/mapreduce/hadoop-mapreduce-examples-$MY_HADOOP_VERSION.jar" pi -Dmapreduce.job.queuename=sample_queue 1 1000;

      fair-scheduler.xml queue config (excerpt):

       

       <queue name="sample_queue">
          <minResources>10000 mb,0vcores</minResources>
          <maxResources>90000 mb,0vcores</maxResources>
          <maxRunningApps>50</maxRunningApps>
          <maxAMShare>-1.0f</maxAMShare>
          <weight>2.0</weight>
          <schedulingPolicy>fair</schedulingPolicy>
        </queue>
      

      Diagnostic message from the web UI: 

      Wed May 02 06:35:57 -0700 2018] Application is added to the scheduler and is not yet activated. (Resource request: <memory:1536, vCores:1> exceeds current queue or its parents maximum resource allowed).

        Attachments

        1. YARN-8248-014.patch
          40 kB
          Szilard Nemeth
        2. YARN-8248-013.patch
          40 kB
          Szilard Nemeth
        3. YARN-8248-012.patch
          40 kB
          Szilard Nemeth
        4. YARN-8248-011.patch
          40 kB
          Szilard Nemeth
        5. YARN-8248-010.patch
          41 kB
          Szilard Nemeth
        6. YARN-8248-009.patch
          41 kB
          Szilard Nemeth
        7. YARN-8248-008.patch
          41 kB
          Szilard Nemeth
        8. YARN-8248-007.patch
          34 kB
          Szilard Nemeth
        9. YARN-8248-006.patch
          22 kB
          Szilard Nemeth
        10. YARN-8248-005.patch
          27 kB
          Szilard Nemeth
        11. YARN-8248-004.patch
          28 kB
          Szilard Nemeth
        12. YARN-8248-003.patch
          28 kB
          Szilard Nemeth
        13. YARN-8248-002.patch
          15 kB
          Szilard Nemeth
        14. YARN-8248-001.patch
          15 kB
          Szilard Nemeth

          Issue Links

            Activity

              People

              • Assignee:
                snemeth Szilard Nemeth
                Reporter:
                snemeth Szilard Nemeth
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: