[MESOS-8778] Fatal error in `DRFSorter::unallocated()` in `SharedPersistentVolumeRescindOnDestroy` test.

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: allocation, master
• Labels: flaky flaky-test

Environment

Centos 6 SSL (internal CI)
asf/master-99c73e0c (12-Apr-2018 09:17:26)

• Sprint: Resource Mgmt R8 Sprint 35, Resource Mgmt R9 Sprint 37, Resource Mgmt RI10 Sp 38, Resource Mgmt RI10 Sp 39
• Story Points: 2
• Assignee: Meng Zhu
• Reporter: Andrei Budnik
• Created: 12/Apr/18 11:27
• Updated: 13/Feb/19 23:34

Description

09:52:15 F0412 09:52:12.391731 23764 sorter.hpp:369] Check failed: scalarQuantities.contains(quantitiesToRemove) cpus:1; mem:128 does not contain cpus:1; mem:128; disk(reservations: [(STATIC,default-role)]):4096
09:52:15 *** Check failure stack trace: ***
09:52:15     @     0x7ff0dfbf5abd  google::LogMessage::Fail()
09:52:15     @     0x7ff0dfbf790d  google::LogMessage::SendToLog()
09:52:15     @     0x7ff0dfbf56a3  google::LogMessage::Flush()
09:52:15     @     0x7ff0dfbf8309  google::LogMessageFatal::~LogMessageFatal()
09:52:15     @     0x7ff0dee93035  mesos::internal::master::allocator::DRFSorter::unallocated()
09:52:15     @     0x7ff0dee69e91  mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::untrackAllocatedResources()
09:52:15     @     0x7ff0dee707a5  mesos::internal::master::allocator::internal::HierarchicalAllocatorProcess::recoverResources()
09:52:15     @     0x7ff0dfb46661  process::ProcessBase::consume()
09:52:15     @     0x7ff0dfb5f8ca  process::ProcessManager::resume()
09:52:15     @     0x7ff0dfb63346  _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
09:52:15     @     0x7ff0dfd1f2e0  execute_native_thread_routine
09:52:15     @     0x7ff0dcd13aa1  start_thread
09:52:15     @     0x7ff0dc0b8bcd  clone

Observed this failure in internal CI for test

DiskResource/PersistentVolumeTest.SharedPersistentVolumeRescindOnDestroy/1

Issue Links

• is caused by: MESOS-8850 Race between master and allocator when destroying shared volume could lead to sorter check failure.