[MESOS-9267] Mesos agent crashes when CNI network is not configured but used.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.2, 1.5.1, 1.6.1, 1.7.0
• Fix Version/s: 1.5.2, 1.6.2, 1.7.1

Environment

Mesos 1.7.0

Marathon 1.7.111

Ubuntu 16.04

• Target Version/s: 1.5.2, 1.6.2, 1.7.1
• Assignee: Jie Yu
• Reporter: z s
• Created: 26/Sep/18 17:02
• Updated: 07/May/19 22:29
• Resolved: 27/Sep/18 22:45

Description

I'm running Mesos 1.7.0 with Marathon 1.7.111 using the Mesos Universal Container Runtime to manage a Docker container. When I set the network mode to "container/bridge" the executor crashing instantly whenever the task is scheduled.

Here are the syslogs

 

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: I0926 16:49:58.604806 46045 containerizer.cpp:3116] Transitioning the state of container a29fd74a-a361-4b3e-8763-5f2cef77380d from PROVISIONING to PREPARING

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: mesos-agent: ../../3rdparty/stout/include/stout/option.hpp:118: const T& Option<T>::get() const & [with T = std::__cxx11::basic_string<char>]: Assertion `isSome()' failed.

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: *** Aborted at 1537980598 (unix time) try "date -d @1537980598" if you are using GNU date ***

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: PC: @     0x7f29695df428 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: *** SIGABRT (@0xb3d1) received by PID 46033 (TID 0x7f295b0ee700) from PID 46033; stack trace: ***

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f2969985390 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695df428 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695e102a (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695d7bd7 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695d7c82 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c63c964 mesos::internal::slave::NetworkCniIsolatorProcess::getNetworkConfigJSON()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c63d3e5 mesos::internal::slave::NetworkCniIsolatorProcess::prepare()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c3700c6 _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchI6OptionIN5mesos5slave19ContainerLaunchInfoEENSB_8internal5slave20MesosIsolatorProcessERKNSB_11ContainerIDERKNSC_15ContainerConfigESK_SN_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSS_FSQ_T1_T2_EOT3_OT4_EUlSt10unique_ptrINS1_7PromiseISE_EESt14default_deleteIS16_EEOSI_OSL_S3_E_IS19_SI_SL_St12_PlaceholderILi1EEEEEEclEOS3_

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce67da1 process::ProcessBase::consume()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce89efa process::ProcessManager::resume()

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce8dc46 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f2969e5fc80 (unknown)

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296997b6ba start_thread

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29696b141d (unknown)

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Main process exited, code=killed, status=6/ABRT

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Unit entered failed state.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Failed with result 'signal'.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Service hold-off time over, scheduling restart.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Stopped Mesos Agent Service.

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Started Mesos Agent Service.

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947785 46255 main.cpp:349] Build: 2018-09-21 14:54:37 by ubuntu

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947849 46255 main.cpp:350] Version: 1.7.0

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947855 46255 main.cpp:353] Git tag: 1.7.0

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947860 46255 main.cpp:357] Git SHA: 8419b870c571ac11825c883fa20ea3b7d4348d34

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.952015 46255 systemd.cpp:240] systemd version `229` detected

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.952047 46255 main.cpp:452] Initializing systemd state

Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Created slice Mesos Executors Slice.
Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.955677 46255 systemd.cpp:328] Started systemd slice `mesos_executors.slice`

Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.956717 46255 resolver.cpp:69] Creating default secret resolver

 

Issue Links

• is duplicated by: MESOS-8972 when choose docker image use user network all mesos agent crash
• is duplicated by: MESOS-8769 Agent crashes when CNI config not defined