[MESOS-5741] Quota + reserved resources is overly conservative

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: mesosphere multitenancy
• Assignee: Unassigned
• Reporter: Neil Conway
• Created: 29/Jun/16 09:10
• Updated: 12/Jul/17 21:05

Description

Consider this scenario:

• Cluster has 10 CPUs total. 8 CPUs are reserved for role X, 2 CPUs are unreserved.
• Role X has a quota for 4 CPUs, but has only been allocated 2 CPUs (e.g., because it has declined an offer for the other 2 CPUs). The CPUs it has been allocated come from the reserved resources, so there are 6 reserved CPUs and 2 unreserved CPUs available.
• That means 6 CPUs should be offered as non-quota resources. However, which 6 CPUs should be offered – the 6 reserved CPUs, or 4 reserved CPUs and 2 unreserved CPUs?

The current quota allocation logic appears to always offer the 6 reserved CPUs. This is unfortunate, because frameworks in other roles won't be able to use those resources. The reason for this behavior is:

  Resources remainingClusterResources = roleSorter->totalScalarQuantities();
  foreachkey (const string& role, activeRoles) {
    remainingClusterResources -= roleSorter->allocationScalarQuantities(role);
  }

remainingClusterResources may have a role set (although dynamically reserved resources will have been converted into effectively static reservations).

  Resources unallocatedQuotaResources;
  foreachpair (const string& name, const Quota& quota, quotas) {
    // Compute the amount of quota that the role does not have allocated.
    //
    // NOTE: Revocable resources are excluded in `quotaRoleSorter`.
    // NOTE: Only scalars are considered for quota.
    Resources allocated = getQuotaRoleAllocatedResources(name);
    const Resources required = quota.info.guarantee();
    unallocatedQuotaResources += (required - allocated);
  }

unallocatedQuotaResources will not have role set, per the implementation of getQuotaRoleAllocatedResources.

remainingClusterResources -= unallocatedQuotaResources;

This means that only unreserved resources will be subtracted from remainingClusterResources. At best this is sub-optimal, because it seems better to lay-away resources for role X that are already reserved for X. I don't think it will result in violating quota guarantees, though.