Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
To unreserve the resource, the current prescribed workflow is to
1. Wait for offer with the reserved resource after the scheduler/role is done using it.
2. Call unreserve on this resource/offer.
3. Done
However this is not reliable if:
1. Master fails to receive the call: this will result in the reserved resources to be offered to the role again, at least there is some signal in this case.
2. Master processes the call but the slave fails to receive the CheckpointResourcesMessage -> then if the master fails over, the slave will reregister with the resource still reserved -> inconsistency here.
3. Master and slave both have processed the call and the resource is unreserved, there is no guarantee that the role unreserving it would receive the offer back. Even if it receives an offer and if the reserved resource is fungible, it cannot distinguish between a resource that is newly unreserved or that is additional resource which is just freed up.
If the framework doesn't go away, it can facilitate the reconciliation but if it wants to terminate, the question is when can it?
The best strategy right now seems to be to for the stopping framework to wait (with a timeout) for the offer to come back after the unreserve call for some verification that's not bulletproof and leave the rest to the operator.
We should improve the reliability of unreserve operations.
Attachments
Issue Links
- is related to
-
MESOS-3826 Add an optional unique identifier for resource reservations
-
- Accepted
-
-
MESOS-3746 Consider introducing a mechanism to provide feedback on offer operations
-
- Resolved
-