[MESOS-6482] Master check failure when marking an agent unreachable

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.1.0
• Component/s: master
• Target Version/s: 1.1.0
• Assignee: Unassigned
• Reporter: Yan Xu
• Created: 25/Oct/16 22:41
• Updated: 28/Oct/16 00:10
• Resolved: 28/Oct/16 00:10

Description

I1025 16:34:55.423038 44118 master.cpp:6006] Marked agent 8e219f7a-06c1-4009-9440-1a33b3e39be5-S473 (x.y.z.com) unreachable: health check timed out
F1025 16:34:55.423632 44118 master.cpp:6036] Check failed: frameworks.recovered.contains(frameworkId) 

Both the master and the agent are on 1.1.

the context

  foreachkey (const FrameworkID& frameworkId, utils::copy(slave->tasks)) {
    Framework* framework = getFramework(frameworkId);

    // If the framework has not yet re-registered after master failover,
    // its FrameworkInfo will be in the `recovered` collection. Note that
    // if the master knows about a task, its FrameworkInfo must appear in
    // either the `registered` or `recovered` collections.
    FrameworkInfo frameworkInfo;

    if (framework == nullptr) {
      CHECK(frameworks.recovered.contains(frameworkId));
      frameworkInfo = frameworks.recovered[frameworkId];
    } else {
      frameworkInfo = framework->info;
    }

    ...

Issue Links

• is broken by: MESOS-4975 mesos::internal::master::Slave::tasks can grow unboundedly
• relates to: MESOS-6483 Check failure when a 1.1 master marking a 0.28 agent as unreachable