Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8248

Job hangs when a job requests a resource that its queue does not have

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.2.0
    • fairscheduler
    • None
    • Reviewed

    Description

      Job hangs when mapreduce.job.queuename is specified and the queue has 0 of any resource (vcores / memory / other)

      In this scenario, the job should be immediately rejected upon submission since the specified queue cannot serve the resource needs of the submitted job.

       

      Command to run:

      bin/yarn jar "./share/hadoop/mapreduce/hadoop-mapreduce-examples-$MY_HADOOP_VERSION.jar" pi -Dmapreduce.job.queuename=sample_queue 1 1000;

      fair-scheduler.xml queue config (excerpt):

       

       <queue name="sample_queue">
          <minResources>10000 mb,0vcores</minResources>
          <maxResources>90000 mb,0vcores</maxResources>
          <maxRunningApps>50</maxRunningApps>
          <maxAMShare>-1.0f</maxAMShare>
          <weight>2.0</weight>
          <schedulingPolicy>fair</schedulingPolicy>
        </queue>
      

      Diagnostic message from the web UI: 

      Wed May 02 06:35:57 -0700 2018] Application is added to the scheduler and is not yet activated. (Resource request: <memory:1536, vCores:1> exceeds current queue or its parents maximum resource allowed).

      Attachments

        1. YARN-8248-001.patch
          15 kB
          Szilard Nemeth
        2. YARN-8248-002.patch
          15 kB
          Szilard Nemeth
        3. YARN-8248-003.patch
          28 kB
          Szilard Nemeth
        4. YARN-8248-004.patch
          28 kB
          Szilard Nemeth
        5. YARN-8248-005.patch
          27 kB
          Szilard Nemeth
        6. YARN-8248-006.patch
          22 kB
          Szilard Nemeth
        7. YARN-8248-007.patch
          34 kB
          Szilard Nemeth
        8. YARN-8248-008.patch
          41 kB
          Szilard Nemeth
        9. YARN-8248-009.patch
          41 kB
          Szilard Nemeth
        10. YARN-8248-010.patch
          41 kB
          Szilard Nemeth
        11. YARN-8248-011.patch
          40 kB
          Szilard Nemeth
        12. YARN-8248-012.patch
          40 kB
          Szilard Nemeth
        13. YARN-8248-013.patch
          40 kB
          Szilard Nemeth
        14. YARN-8248-014.patch
          40 kB
          Szilard Nemeth

        Issue Links

          Activity

            People

              snemeth Szilard Nemeth
              snemeth Szilard Nemeth
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: