[MESOS-2818] Pass 'allocated' resources for each executor to the resource estimator.

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.23.0
• Labels: twitter
• Epic Link: Oversubscription
• Sprint: Twitter Q2 Sprint 3, Twitter Mesos Q2 Sprint 5
• Story Points: 3
• Assignee: Jie Yu
• Reporter: Jie Yu
• Created: 05/Jun/15 00:47
• Updated: 15/Jun/15 22:35
• Resolved: 11/Jun/15 18:38

Description

Resource estimator obviously need this information to calculate, say the usage slack. Now the question is how. There are two approaches:

1) Pass in the allocated resources for each executor through the 'oversubscribable()' interface.

2) Let containerizer return total resources allocated for each container when 'usages()' are invoked.

I would suggest to take route (1) for several reasons:

1) Eventually, we'll need to pass in slave's total resources to the resource estimator (so that RE can calculate allocation slack). There is no way that we can get that from containerizer. The slave's total resources keep changing due to dynamic reservation. So we cannot pass in the slave total resources during initialization.

2) The current implementation of usages() might skip some containers if it fails to get statistics for that container (not an error). This will cause in-complete information to the RE.

3) We may want to calculate 'unallocated = total - allocated' so that we can send allocation slack as well. Getting 'total' and 'allocated' from two different components might result in inconsistent value. Remember that 'total' keeps changing due to dynamic reservation.

Issue Links

• blocks: MESOS-2791 Create a FixedResourceEstimator to return fixed amount of oversubscribable resources.
• is duplicated by: MESOS-2741 Exposing Resources along with ResourceStatistics from resource monitor