[MESOS-9365] Windows - GET_CONTAINERS API call causes the Mesos agent to fail

• Type: Bug
• Status: Accepted
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Component/s: agent
• Labels: foundations
• Assignee: Unassigned
• Reporter: Ionut Balutoiu
• Created: 31/Oct/18 11:58
• Updated: 17/Jul/19 16:23

Description

The Mesos agent on Windows crashes abruptly when someone is doing the GET_CONTAINERS API call.

Snippet from the log file after the API call:

I1025 12:06:50.014464 7296 http.cpp:2072] Processing GET_CONTAINERS call
F1025 12:06:50.185472 7296 evolve.cpp:736] Check failed: '::protobuf::parse<v1::ResourceStatistics>(resource_statistics.get())' Must be SOME: Missing required fields: timestamp 
*** Check failure stack trace: ***
# Child-SP          RetAddr           Call Site
00 00000044`37cfb3a8 00007ffa`a9a4834c ntdll!NtWaitForMultipleObjects+0x14
01 00000044`37cfb3b0 00007ffa`a9a47656 ntdll!WerpWaitForCrashReporting+0xa8
02 00000044`37cfb430 00007ffa`a9a46e1d ntdll!RtlReportExceptionHelper+0x33e
03 00000044`37cfb500 00007ffa`a9a64d95 ntdll!RtlReportException+0x9d
04 00000044`37cfb580 00007ffa`a99fab36 ntdll!RtlReportCriticalFailure$filt$0+0x33
05 00000044`37cfb5b0 00007ffa`a9a09aee ntdll!_C_specific_handler+0x96
06 00000044`37cfb620 00007ffa`a9a0ed2d ntdll!_GSHandlerCheck_SEH+0x6a
07 00000044`37cfb650 00007ffa`a9976c86 ntdll!RtlpExecuteHandlerForException+0xd
08 00000044`37cfb680 00007ffa`a99752ca ntdll!RtlDispatchException+0x3c6
09 00000044`37cfbd80 00007ffa`a9a64d3b ntdll!RtlRaiseException+0x31a
0a 00000044`37cfc560 00007ffa`a9a6c806 ntdll!RtlReportCriticalFailure+0x97
0b 00000044`37cfc670 00007ffa`a9a6cad1 ntdll!RtlpHeapHandleError+0x12
0c 00000044`37cfc6a0 00007ffa`a9a09a55 ntdll!RtlpHpHeapHandleError+0x5d
0d 00000044`37cfc6d0 00007ffa`a9a17347 ntdll!RtlpLogHeapFailure+0x45
0e 00000044`37cfc700 00007ffa`a6c6d92b ntdll!RtlFreeHeap+0x97127
0f 00000044`37cfc7a0 00007ffa`a6c6d7da ucrtbase!_free_base+0x1b
10 00000044`37cfc7d0 00007ffa`a370c321 ucrtbase!__crt_state_management::wrapped_invoke<void (_cdecl*)(void * _ptr64),void * __ptr64,void>+0x2a
11 00000044`37cfc800 00007ffa`a3718972 dbghelp!DIA::`scalar deleting destructor'+0x81
12 00000044`37cfc840 00007ffa`a3736de6 dbghelp!diaRelease+0x26
13 00000044`37cfc870 00007ffa`a372aea4 dbghelp!FreeModuleEntry+0x1ae
14 00000044`37cfc8a0 00007ffa`a372bd5f dbghelp!lSymCleanup+0x84
15 00000044`37cfc900 00007ff6`ccc25b47 dbghelp!SymCleanup+0x1f
16 00000044`37cfc930 00007ff6`ccc25035 mesos_agent!std::_Func_impl_no_alloc<<lambda_7946926b5f7d2a4f51979db6e8cea4e0>,Try<Nothing,Error>,flags::FlagsBase *,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &>::_Do_call+0x97
17 00000044`37cfd2d0 00007ff6`ccc25117 mesos_agent!std::_Func_impl_no_alloc<<lambda_541f99c14a5c3a22c2d32bcf7b7a6ec2>,Option<Error>,flags::FlagsBase const &>::_Do_call+0x25 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\functional @ 15732480] 
18 00000044`37cfdcc0 00007ff6`ccc237cd mesos_agent!std::_Func_impl_no_alloc<<lambda_5681e293f27e1c45e4d7b18a16015afe>,Option<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,flags::FlagsBase const &>::_Do_call+0x77
19 00000044`37cfdcf0 00007ff6`ccc21685 mesos_agent!std::vector<flags::Name,std::allocator<flags::Name> >::_Destroy+0x1d [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\vector @ 1848] 
1a 00000044`37cfddc0 00007ff6`ccc20da2 mesos_agent!std::_Func_impl_no_alloc<<lambda_46cfdb4aa393cccbb30241109933da08>,Try<Nothing,Error>,flags::FlagsBase *,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &>::_Delete_this+0x35 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\functional @ 1243] 
1b 00000044`37cfde00 00007ff6`cc9a358b mesos_agent!std::_Func_impl_no_alloc<<lambda_09eef18ca52599782eac8ff0104f0c99>,Option<Error>,flags::FlagsBase const &>::_Delete_this+0x52
1c 00000044`37cfde40 00007ff6`cc9a5806 mesos_agent!Option<Option<std::vector<process::Owned<mesos::ObjectApprover>,std::allocator<process::Owned<mesos::ObjectApprover> > > > >::operator=+0x1b [d:\dcos\mesos\mesos\3rdparty\stout\include\stout\option.hpp @ 101] 
1d 00000044`37cfdf50 00007ff6`cca0ee3f mesos_agent!google::protobuf::internal::RepeatedPtrIterator<mesos::Attribute const >::operator!=+0x56
1e 00000044`37cfe660 00007ff6`cca1c492 mesos_agent!std::_List_node<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Hook >,void >::_Freenode0<std::allocator<std::_List_node<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Hook >,void > > >+0x6f [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\list @ 374] 
1f 00000044`37cfe8c0 00007ff6`cca00b35 mesos_agent!process::Future<mesos::DockerTaskExecutorPrepareInfo>::onDiscard<std::_Binder<std::_Unforced,void (__cdecl*)(process::WeakFuture<std::vector<Option<mesos::DockerTaskExecutorPrepareInfo>,std::allocator<Option<mesos::DockerTaskExecutorPrepareInfo> > > >),process::WeakFuture<std::vector<Option<mesos::DockerTaskExecutorPrepareInfo>,std::allocator<Option<mesos::DockerTaskExecutorPrepareInfo> > > > > >+0xa2 [d:\dcos\mesos\mesos\3rdparty\libprocess\include\process\future.hpp @ 351] 
20 00000044`37cfea00 00007ff6`cc745cf8 mesos_agent!mesos::internal::HookManager::slaveAttributesDecorator+0x425
21 00000044`37cfea70 00007ff6`cc7316a4 mesos_agent!ILT+281840(??0?$FunctionClosure1PEAVLazyDescriptorinternalprotobufgoogleinternalprotobufgoogleQEAAP6AXPEAVLazyDescriptor+0x3
22 00000044`37cfeaa0 00007ff6`cc9e86cc mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
23 00000044`37cfeb00 00007ff6`cc7316a4 mesos_agent!std::_Hash<std::_Umap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Value_Ranges,std::_Uhash_compare<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::hash<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::equal_to<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,mesos::Value_Ranges> >,0> >::erase+0x12c [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xhash @ 607] 
24 00000044`37cfebf0 00007ff6`cc9e86bd mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
25 00000044`37cfec50 00007ff6`cc7316a4 mesos_agent!std::_Hash<std::_Umap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Value_Ranges,std::_Uhash_compare<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::hash<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::equal_to<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,mesos::Value_Ranges> >,0> >::erase+0x11d [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xhash @ 607] 
26 00000044`37cfed40 00007ff6`cc9e86bd mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
27 00000044`37cfeda0 00007ff6`cc7316a4 mesos_agent!std::_Hash<std::_Umap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Value_Ranges,std::_Uhash_compare<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::hash<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::equal_to<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,mesos::Value_Ranges> >,0> >::erase+0x11d [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xhash @ 607] 
28 00000044`37cfee90 00007ff6`cc9e86bd mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
29 00000044`37cfeef0 00007ff6`cc7316a4 mesos_agent!std::_Hash<std::_Umap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Value_Ranges,std::_Uhash_compare<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::hash<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::equal_to<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,mesos::Value_Ranges> >,0> >::erase+0x11d [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xhash @ 607] 
2a 00000044`37cfefe0 00007ff6`cc9e86bd mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
2b 00000044`37cff040 00007ff6`cc9fbf37 mesos_agent!std::_Hash<std::_Umap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,mesos::Value_Ranges,std::_Uhash_compare<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::hash<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::equal_to<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,mesos::Value_Ranges> >,0> >::erase+0x11d [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xhash @ 607] 
2c 00000044`37cff130 00007ff6`cca33677 mesos_agent!rapidjson::internal::u64toa+0x167 [d:\dcos\mesos\3rdparty\rapidjson-1.1.0\src\rapidjson-1.1.0\include\rapidjson\internal\itoa.h @ 144] 
2d 00000044`37cff1c0 00007ff6`cca01863 mesos_agent!std::_Vector_alloc<std::_Vec_base_types<Option<mesos::DockerTaskExecutorPrepareInfo>,std::allocator<Option<mesos::DockerTaskExecutorPrepareInfo> > > >::_Get_data+0x7 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\vector @ 554] 
2e 00000044`37cff320 00007ff6`cc745cf8 mesos_agent!std::_Compressed_pair<std::allocator<lambda::CallableOnce<void _cdecl(process::Future<mesos::DockerTaskExecutorPrepareInfo> const &)> >,std::_Vector_val<std::_Simple_types<lambda::CallableOnce<void _cdecl(process::Future<mesos::DockerTaskExecutorPrepareInfo> const &)> > >,1>::_Compressed_pair<std::allocator<lambda::CallableOnce<void _cdecl(process::Future<mesos::DockerTaskExecutorPrepareInfo> const &)> >,std::_Vector_val<std::_Simple_types<lambda::CallableOnce<void _cdecl(process::Future<mesos::DockerTaskExecutorPrepareInfo> const &)> > >,1><>+0x13 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xutility @ 277] 
2f 00000044`37cff390 00007ff6`cc7316a4 mesos_agent!ILT+281840(??0?$FunctionClosure1PEAVLazyDescriptorinternalprotobufgoogleinternalprotobufgoogleQEAAP6AXPEAVLazyDescriptor+0x3
30 00000044`37cff3c0 00007ff6`cc7ce2a3 mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
31 00000044`37cff420 00007ff6`cc9fc5f7 mesos_agent!ILT+840350(??1?$unique_ptrUCallable?$CallableOnce$$A6AXAEBV?$FuturePEAUMessageEventprocessprocessZlambdaU?$default_deleteUCallable?$CallableOnce$$A6AXAEBV?$FuturePEAUMessageEventprocessprocessZlambdastdstdQEAAXZ)
32 00000044`37cff520 00007ff6`cca34557 mesos_agent!rapidjson::internal::u64toa+0x827 [d:\dcos\mesos\3rdparty\rapidjson-1.1.0\src\rapidjson-1.1.0\include\rapidjson\internal\itoa.h @ 216] 
33 00000044`37cff5b0 00007ff6`cca019b3 mesos_agent!std::vector<Option<mesos::DockerTaskExecutorPrepareInfo>,std::allocator<Option<mesos::DockerTaskExecutorPrepareInfo> > >::_Has_unused_capacity+0x47 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\vector @ 1726] 
34 00000044`37cff710 00007ff6`cc745cf8 mesos_agent!std::_Compressed_pair<std::allocator<process::Future<Option<mesos::DockerTaskExecutorPrepareInfo> > >,std::_Vector_val<std::_Simple_types<process::Future<Option<mesos::DockerTaskExecutorPrepareInfo> > > >,1>::_Compressed_pair<std::allocator<process::Future<Option<mesos::DockerTaskExecutorPrepareInfo> > >,std::_Vector_val<std::_Simple_types<process::Future<Option<mesos::DockerTaskExecutorPrepareInfo> > > >,1><>+0x43 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xutility @ 279] 
35 00000044`37cff780 00007ff6`cc7316a4 mesos_agent!ILT+281840(??0?$FunctionClosure1PEAVLazyDescriptorinternalprotobufgoogleinternalprotobufgoogleQEAAP6AXPEAVLazyDescriptor+0x3
36 00000044`37cff7b0 00007ff6`cc7cee1e mesos_agent!ILT+198300(??$_Uninitialized_movePEAV?$_List_unchecked_iteratorV?$_List_valU?$_List_simple_typesU?$pairQEBVDescriptorprotobufgooglePEBVMessage+0x3
37 00000044`37cff810 00007ff6`cc7ed443 mesos_agent!ILT+843285(?clear_has_key?$MapEntryImplVGetCapacityRequest_ParametersEntry_DoNotUsev0csiVMessageprotobufgoogleV?$basic_stringDU?$char_traitsDstdV?$allocatorD+0x4
38 00000044`37cff900 00007ff6`ccba07f6 mesos_agent!ILT+967740(?GetMetadataRequestmesosUEBA?AUMetadataprotobufgoogleXZ)+0x2
39 00000044`37cff950 00007ff6`ccbb7fe5 mesos_agent!std::weak_ptr<process::Future<Try<std::tuple<unsigned __int64,std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,mesos::internal::FilesError> >::Data>::lock+0x46 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\memory @ 1996] 
3a 00000044`37cff9a0 00007ff6`ccb9a04c mesos_agent!std::function<Try<Nothing,Error> __cdecl(flags::FlagsBase *,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &)>::operator=<<lambda_12d7f8fae9e986fc019eb97a0dbc4d01>,void>+0x35 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\functional @ 1563] 
3b 00000044`37cffbd0 00007ff6`ccb982f9 mesos_agent!Try<std::list<mesos::FileInfo,std::allocator<mesos::FileInfo> >,mesos::internal::FilesError>::error+0xbc
3c 00000044`37cffc10 00007ffa`a6c7c4ce mesos_agent!std::_Hash<std::_Umap_traits<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::function<process::Future<bool> _cdecl(Option<process::http::authentication::Principal> const &)>,std::_Uhash_compare<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,std::hash<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >,std::equal_to<std::basic_string<char,std::char_traits<char>,std::allocator<char> > > >,std::allocator<std::pair<std::basic_string<char,std::char_traits<char>,std::allocator<char> > const ,std::function<process::Future<bool> _cdecl(Option<process::http::authentication::Principal> const &)> > >,0> >::equal_range+0x199 [c:\program files (x86)\microsoft visual studio\2017\community\vc\tools\msvc\14.15.26726\include\xhash @ 733] 
3d 00000044`37cffc40 00007ffa`a9893034 ucrtbase!thread_start<unsigned int (_cdecl*)(void * _ptr64)>+0x3e
3e 00000044`37cffc70 00007ffa`a99e1461 kernel32!BaseThreadInitThunk+0x14
3f 00000044`37cffca0 00000000`00000000 ntdll!RtlUserThreadStart+0x21

Process "c:\opt\mesosphere\bin\mesos-agent.exe" failed with exit code: 3221226505

The mesos-agent.exe is a binary built against this codebase from the Apache master branch.