[YARN-177] CapacityScheduler - adding a queue while the RM is running has wacky results

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.3
• Fix Version/s: 2.0.3-alpha, 0.23.5
• Component/s: capacityscheduler
• Assignee: Arun Murthy
• Reporter: Thomas Graves
• Created: 22/Oct/12 01:07
• Updated: 03/Sep/14 23:18
• Resolved: 24/Oct/12 14:08

Description

Adding a queue to the capacity scheduler while the RM is running and then running a job in the queue added results in very strange behavior. The cluster Total Memory can either decrease or increase. We had a cluster where total memory decreased to almost 1/6th the capacity. Running on a small test cluster resulted in the capacity going up by simply adding a queue and running wordcount.

Looking at the RM logs, used memory can go negative but other logs show the number positive:

2012-10-21 22:56:44,796 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.0375 absoluteUsedCapacity=0.0375 used=memory: 7680 cluster=memory: 204800

2012-10-21 22:56:45,831 [ResourceManager Event Processor] INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=-0.0225 absoluteUsedCapacity=-0.0225 used=memory: -4608 cluster=memory: 204800