Hadoop YARN
  1. Hadoop YARN
  2. YARN-177

CapacityScheduler - adding a queue while the RM is running has wacky results

    Details

      Description

      Adding a queue to the capacity scheduler while the RM is running and then running a job in the queue added results in very strange behavior. The cluster Total Memory can either decrease or increase. We had a cluster where total memory decreased to almost 1/6th the capacity. Running on a small test cluster resulted in the capacity going up by simply adding a queue and running wordcount.

      Looking at the RM logs, used memory can go negative but other logs show the number positive:

      2012-10-21 22:56:44,796 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.0375 absoluteUsedCapacity=0.0375 used=memory: 7680 cluster=memory: 204800

      2012-10-21 22:56:45,831 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=-0.0225 absoluteUsedCapacity=-0.0225 used=memory: -4608 cluster=memory: 204800

      1. YARN-177.patch
        16 kB
        Arun C Murthy
      2. YARN-177.patch
        11 kB
        Arun C Murthy
      3. YARN-177.patch
        8 kB
        Thomas Graves
      4. YARN-177.patch
        8 kB
        Arun C Murthy

        Activity

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development