[YARN-4045] Negative avaialbleMB is being reported for root queue.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.7.1
• Assignee: Unassigned
• Reporter: Rushabh Shah
• Created: 11/Aug/15 17:11
• Updated: 13/Feb/17 00:27
• Resolved: 13/Feb/17 00:27

Description

We recently deployed 2.7 in one of our cluster.
We are seeing negative availableMB being reported for queue=root.
This is from the jmx output:

<clusterMetrics>
    ...
    <availableMB>-163328</availableMB>
    ...
</clusterMetrics>

The following is the RM log:

2015-08-10 14:42:28,280 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:28,404 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:30,913 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:30,913 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:33,093 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:33,093 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:35,548 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:35,549 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:39,088 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:39,089 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:39,338 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:39,339 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:39,757 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:39,758 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:43,056 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:43,070 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:44,486 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:44,487 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:44,886 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:44,886 [ResourceManager Event Processor] INFO capacity.ParentQueue: assignedContainer queue=root usedCapacity=1.0032743 absoluteUsedCapacity=1.0032743 used=<memory:5334016, vCores:6212> cluster=<memory:5316608, vCores:28320>
2015-08-10 14:42:47,401 [ResourceManager Event Processor] INFO capacity.ParentQueue: completedContainer queue=root usedCapacity=1.0029854 absoluteUsedCapacity=1.0029854 used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>

used=<memory:5332480, vCores:6202> cluster=<memory:5316608, vCores:28320>

For root queue, usedCapacity is more than totalCapacity

Issue Links

• duplicates: YARN-3933 FairScheduler: Multiple calls to completedContainer are not safe
• relates to: YARN-4067 available resource could be set negative
• relates to: YARN-4089 Race condition when calling AbstractYarnScheduler.completedContainer.