[MESOS-9786] Race between two REMOVE_QUOTA calls crashes the master.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.3, 1.6.2, 1.7.2, 1.8.0, 1.9.0
• Fix Version/s: 1.5.4, 1.6.3, 1.7.3, 1.8.1, 1.9.0
• Labels: resource-management
• Sprint: Resource Mgmt: RI14 Sp 46
• Story Points: 2
• Assignee: Meng Zhu
• Reporter: Andrei Sekretenko
• Created: 15/May/19 14:25
• Updated: 22/May/19 16:29
• Resolved: 16/May/19 15:45

Description

The existence of the quota in the master is validated here:
https://github.com/apache/mesos/blob/a9a2acabd03181865055b77cf81e7bb310b236d6/src/master/quota_handler.cpp#L700

Then the quota is removed from master in a deferred method call:
https://github.com/apache/mesos/blob/a9a2acabd03181865055b77cf81e7bb310b236d6/src/master/quota_handler.cpp#L744

And then removed from allocator in another deferred call:
https://github.com/apache/mesos/blob/a9a2acabd03181865055b77cf81e7bb310b236d6/src/master/quota_handler.cpp#L753

So, there is a race between two simultaneous REMOVE_QUOTA calls.

We observe this race on a heavily loaded cluster. Currently we suspect that the client retries the call (due to the call being not processed for a long time),  and this triggers the race.