Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9267

Mesos agent crashes when CNI network is not configured but used.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.2, 1.5.1, 1.6.1, 1.7.0
    • Fix Version/s: 1.5.2, 1.6.2, 1.7.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      Mesos 1.7.0

      Marathon 1.7.111

      Ubuntu 16.04

      Description

      I'm running Mesos 1.7.0 with Marathon 1.7.111 using the Mesos Universal Container Runtime to manage a Docker container. When I set the network mode to "container/bridge" the executor crashing instantly whenever the task is scheduled.

      Here are the syslogs

       

      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: I0926 16:49:58.604806 46045 containerizer.cpp:3116] Transitioning the state of container a29fd74a-a361-4b3e-8763-5f2cef77380d from PROVISIONING to PREPARING
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: mesos-agent: ../../3rdparty/stout/include/stout/option.hpp:118: const T& Option<T>::get() const & [with T = std::__cxx11::basic_string<char>]: Assertion `isSome()' failed.
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: *** Aborted at 1537980598 (unix time) try "date -d @1537980598" if you are using GNU date ***
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: PC: @     0x7f29695df428 (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]: *** SIGABRT (@0xb3d1) received by PID 46033 (TID 0x7f295b0ee700) from PID 46033; stack trace: ***
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f2969985390 (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695df428 (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695e102a (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695d7bd7 (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29695d7c82 (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c63c964 mesos::internal::slave::NetworkCniIsolatorProcess::getNetworkConfigJSON()
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c63d3e5 mesos::internal::slave::NetworkCniIsolatorProcess::prepare()
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296c3700c6 _ZNO6lambda12CallableOnceIFvPN7process11ProcessBaseEEE10CallableFnINS_8internal7PartialIZNS1_8dispatchI6OptionIN5mesos5slave19ContainerLaunchInfoEENSB_8internal5slave20MesosIsolatorProcessERKNSB_11ContainerIDERKNSC_15ContainerConfigESK_SN_EENS1_6FutureIT_EERKNS1_3PIDIT0_EEMSS_FSQ_T1_T2_EOT3_OT4_EUlSt10unique_ptrINS1_7PromiseISE_EESt14default_deleteIS16_EEOSI_OSL_S3_E_IS19_SI_SL_St12_PlaceholderILi1EEEEEEclEOS3_
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce67da1 process::ProcessBase::consume()
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce89efa process::ProcessManager::resume()
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296ce8dc46 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUlvE_vEEE6_M_runEv
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f2969e5fc80 (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f296997b6ba start_thread
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46033]:     @     0x7f29696b141d (unknown)
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Main process exited, code=killed, status=6/ABRT
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Unit entered failed state.
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Failed with result 'signal'.
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: mesos-agent.service: Service hold-off time over, scheduling restart.
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Stopped Mesos Agent Service.
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Started Mesos Agent Service.
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947785 46255 main.cpp:349] Build: 2018-09-21 14:54:37 by ubuntu
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947849 46255 main.cpp:350] Version: 1.7.0
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947855 46255 main.cpp:353] Git tag: 1.7.0
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.947860 46255 main.cpp:357] Git SHA: 8419b870c571ac11825c883fa20ea3b7d4348d34
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.952015 46255 systemd.cpp:240] systemd version `229` detected
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.952047 46255 main.cpp:452] Initializing systemd state
      
      Sep 26 16:49:58 ip-172-27-1-88 systemd[1]: Created slice Mesos Executors Slice.
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.955677 46255 systemd.cpp:328] Started systemd slice `mesos_executors.slice`
      
      Sep 26 16:49:58 ip-172-27-1-88 mesos-agent[46255]: I0926 16:49:58.956717 46255 resolver.cpp:69] Creating default secret resolver
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jieyu Jie Yu
                Reporter:
                dkjs z s
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: