Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10428

Zombie applications in the YARN queue using FAIR + sizebasedweight

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Seeing zombie jobs in the YARN queue that uses FAIR and size based weight ordering policy .

      Detection:

      The YARN UI shows incorrect number of "Num Schedulable Applications".

      Impact:

      The queue has an upper limit of number of running applications, with zombie job, it hits the limit even though the number of running applications is far less than the limit. 

      Workaround:

      **Fail-over and restart Resource Manager process.

      Analysis:

      **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy#

      schedulableEntities` (see attachment). Take application "application_1599157165858_29429" for example, it is still in the  `FairOderingPolicy#schedulableEntities` set, however, if we check the log of resource manager, we can see RM already tried to remove the application:

       

      ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 04:32:19,730 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue (ResourceManager Event Processor): Application removed - appId: application_1599157165858_29429 user: svc_di_data_eng queue: core-data #user-pending-applications: -3 #user-active-applications: 7 #queue-pending-applications: 0 #queue-active-applications: 21

       

      So it appears RM failed to removed the application from the set.

      Attachments

        1. YARN-10428.003.patch
          4 kB
          Andras Gyori
        2. YARN-10428.002.patch
          5 kB
          Guang Yang
        3. YARN-10428.001.patch
          5 kB
          Guang Yang

        Activity

          People

            gandras Andras Gyori
            yguang11 Guang Yang
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: