Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-177

CapacityScheduler - adding a queue while the RM is running has wacky results

    Details

      Description

      Adding a queue to the capacity scheduler while the RM is running and then running a job in the queue added results in very strange behavior. The cluster Total Memory can either decrease or increase. We had a cluster where total memory decreased to almost 1/6th the capacity. Running on a small test cluster resulted in the capacity going up by simply adding a queue and running wordcount.

      Looking at the RM logs, used memory can go negative but other logs show the number positive:

      2012-10-21 22:56:44,796 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.0375 absoluteUsedCapacity=0.0375 used=memory: 7680 cluster=memory: 204800

      2012-10-21 22:56:45,831 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=-0.0225 absoluteUsedCapacity=-0.0225 used=memory: -4608 cluster=memory: 204800

        Attachments

        1. YARN-177.patch
          16 kB
          Arun C Murthy
        2. YARN-177.patch
          11 kB
          Arun C Murthy
        3. YARN-177.patch
          8 kB
          Thomas Graves
        4. YARN-177.patch
          8 kB
          Arun C Murthy

          Activity

            People

            • Assignee:
              acmurthy Arun C Murthy
              Reporter:
              tgraves Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: