[MESOS-8469] Mesos master might drop some events in the operator API stream

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Sprint: Mesosphere Sprint 73
• Story Points: 3
• Assignee: Greg Mann
• Reporter: Vinod Kone
• Shepherd: Vinod Kone
• Created: 20/Jan/18 01:31
• Updated: 22/Feb/18 23:20
• Resolved: 23/Jan/18 00:15

Description

Inside `Master::updateTask`, we call `Subscribers::send` which asynchronously calls `Subscribers::Subscriber::send` on each subscriber.

But the problem is that inside `Subscribers:Subscriber::send` we are looking up the state of the master (e.g., getting Task* and Framework*) which might have changed between `Subscribers::send ` and `Subscribers::Subscriber::send`.

 

For example, if a terminal task received an acknowledgement the task might be removed from master's state, causing us to drop the TASK_UPDATED event.

 

We noticed this in an internal cluster, where a TASK_KILLED update was sent to one subscriber but not the other.

 

 

 

Issue Links

• breaks: MESOS-8602 Subscribers::send incorrectly assumes frameworks are registered