[YARN-4979] FSAppAttempt demand calculation considers demands at multiple locality levels different

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 2.7.2
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: fairscheduler
• Hadoop Flags: Reviewed
• Assignee: Zhihai Xu
• Reporter: Zhihai Xu
• Created: 21/Apr/16 06:16
• Updated: 30/Aug/16 01:02
• Resolved: 23/May/16 21:45

Description

FSAppAttempt adds duplicate ResourceRequest to demand in updateDemand. We should only count ResourceRequest for ResourceRequest.ANY when calculate demand.
Because hasContainerForNode will return false if no container request for ResourceRequest.ANY and both allocateNodeLocal and allocateRackLocal will also decrease the number of containers for ResourceRequest.ANY.
This issue may cause current memory demand overflow(integer) because duplicate requests can be on multiple nodes.

Issue Links

• is related to: YARN-1458 FairScheduler: Zero weight can lead to livelock