[YARN-6831] Miscellaneous refactoring changes of ContainScheduler

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager
• Target Version/s: 3.4.0
• Assignee: Haibo Chen
• Reporter: Haibo Chen
• Created: 17/Jul/17 17:29
• Updated: 09/Apr/20 18:38

Description

While reviewing YARN-6706, Karthik pointed out a few issues for improvment in ContainerScheduler

*Make ResourceUtilizationTracker pluggable. That way, we could use a different tracker when oversubscription is enabled.

*ContainerScheduler
##Why do we need maxOppQueueLength given queuingLimit?
##Is there value in splitting runningContainers into runningGuaranteed and runningOpportunistic?
##getOpportunisticContainersStatus method implementation feels awkward. How about capturing the state in the field here, and have metrics etc. pull from here?
##startContainersFromQueue: Local variable resourcesAvailable is unnecessary

*OpportunisticContainersStatus
##Let us clearly differentiate between allocated, used and utilized. Maybe, we should rename current Used methods to Allocated?
##I prefer either full name Opportunistic (in method) or Opp (shortest name that makes sense). Opport is neither short nor fully descriptive.
##Have we considered folding ContainerQueuingLimit class into this?

We decided to move the issues into this follow up jira to keep YARN-6706 moving forward to unblock oversubscription work.

Sub-Tasks

• Remove runningContainers from ContainerScheduler: Patch Available