[MESOS-6179] HealthCheckTest.ROOT_DOCKER_DockerHealthStatusChange is flaky - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: docker, test
Labels:
- docker
- foundations
- health-check
- mesosphere
- tests
Environment:

CentOS 6, libevent/SSL disabled

Epic Link:
flaky

Description

Observed on our internal CI:

[03:00:48] :	 [Step 10/10] [ RUN      ] HealthCheckTest.ROOT_DOCKER_DockerHealthStatusChange
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.877110 25514 cluster.cpp:157] Creating default 'local' authorizer
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.885874 25514 leveldb.cpp:174] Opened db in 8.637338ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888603 25514 leveldb.cpp:181] Compacted db in 2.710903ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888625 25514 leveldb.cpp:196] Created db iterator in 5044ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888631 25514 leveldb.cpp:202] Seeked to beginning of db in 713ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888638 25514 leveldb.cpp:271] Iterated through 0 keys in the db in 701ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888648 25514 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888767 25531 recover.cpp:451] Starting replica recovery
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.888835 25533 recover.cpp:477] Replica is in EMPTY status
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889119 25529 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(865)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889196 25532 recover.cpp:197] Received a recover response from a replica in EMPTY status
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889443 25535 recover.cpp:568] Updating replica status to STARTING
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889562 25532 master.cpp:380] Master 4227c978-ffc0-453f-b611-353e1665bd0f (ip-172-30-2-21.mesosphere.io) started on 172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889582 25532 master.cpp:382] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/u5lSkR/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/u5lSkR/master" --zk_session_timeout="10secs"
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889734 25532 master.cpp:432] Master only allowing authenticated frameworks to register
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889739 25532 master.cpp:446] Master only allowing authenticated agents to register
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889744 25532 master.cpp:459] Master only allowing authenticated HTTP frameworks to register
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889746 25532 credentials.hpp:37] Loading credentials for authentication from '/tmp/u5lSkR/credentials'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889828 25532 master.cpp:504] Using default 'crammd5' authenticator
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889863 25532 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889895 25532 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889930 25532 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.889967 25532 master.cpp:584] Authorization enabled
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.890024 25530 hierarchical.cpp:149] Initialized hierarchical allocator process
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.890029 25536 whitelist_watcher.cpp:77] No whitelist given
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.890537 25532 master.cpp:1855] Elected as the leading master!
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.890548 25532 master.cpp:1556] Recovering from registrar
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.890602 25530 registrar.cpp:332] Recovering registrar
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.892370 25535 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.881085ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.892385 25535 replica.cpp:320] Persisted replica status to STARTING
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.892428 25535 recover.cpp:477] Replica is in STARTING status
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.892698 25536 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(866)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.892807 25536 recover.cpp:197] Received a recover response from a replica in STARTING status
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.893007 25535 recover.cpp:568] Updating replica status to VOTING
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.895607 25535 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.5434ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.895622 25535 replica.cpp:320] Persisted replica status to VOTING
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.895659 25535 recover.cpp:582] Successfully joined the Paxos group
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.895723 25535 recover.cpp:466] Recover process terminated
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.895901 25536 log.cpp:553] Attempting to start the writer
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.896302 25531 replica.cpp:493] Replica received implicit promise request from __req_res__(867)@172.30.2.21:53211 with proposal 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.898988 25531 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.667146ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.899001 25531 replica.cpp:342] Persisted promised to 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.899206 25529 coordinator.cpp:238] Coordinator attempting to fill missing positions
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.899660 25532 replica.cpp:388] Replica received explicit promise request from __req_res__(868)@172.30.2.21:53211 for position 0 with proposal 2
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.910912 25532 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 11.228654ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.910928 25532 replica.cpp:708] Persisted action NOP at position 0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.911296 25534 replica.cpp:537] Replica received write request for position 0 from __req_res__(869)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.911325 25534 leveldb.cpp:436] Reading position from leveldb took 9832ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.914330 25534 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 2.990437ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.914345 25534 replica.cpp:708] Persisted action NOP at position 0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.914567 25534 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.917343 25534 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 2.760605ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.917357 25534 replica.cpp:708] Persisted action NOP at position 0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.917557 25532 log.cpp:569] Writer started with ending position 0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.917820 25533 leveldb.cpp:436] Reading position from leveldb took 10851ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.918045 25533 registrar.cpp:365] Successfully fetched the registry (0B) in 27.419136ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.918076 25533 registrar.cpp:464] Applied 1 operations in 3094ns; attempting to update the registry
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.918244 25531 log.cpp:577] Attempting to append 205 bytes to the log
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.918287 25531 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.918525 25534 replica.cpp:537] Replica received write request for position 1 from __req_res__(870)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.921182 25534 leveldb.cpp:341] Persisting action (224 bytes) to leveldb took 2.639056ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.921197 25534 replica.cpp:708] Persisted action APPEND at position 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.921438 25536 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924043 25536 leveldb.cpp:341] Persisting action (226 bytes) to leveldb took 2.58637ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924057 25536 replica.cpp:708] Persisted action APPEND at position 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924252 25534 registrar.cpp:509] Successfully updated the registry in 6.160128ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924374 25534 registrar.cpp:395] Successfully recovered registrar
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924381 25532 log.cpp:596] Attempting to truncate the log to 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924523 25535 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924523 25534 master.cpp:1664] Recovered 0 agents from the registry (166B); allowing 10mins for agents to re-register
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924568 25529 hierarchical.cpp:176] Skipping recovery of hierarchical allocator: nothing to recover
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.924836 25533 replica.cpp:537] Replica received write request for position 2 from __req_res__(871)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.927575 25533 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 2.722343ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.927590 25533 replica.cpp:708] Persisted action TRUNCATE at position 2
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.927886 25535 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.931293 25535 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 3.386152ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.931320 25535 leveldb.cpp:399] Deleting ~1 keys from leveldb took 13672ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.931329 25535 replica.cpp:708] Persisted action TRUNCATE at position 2
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.932735 25514 cluster.cpp:436] Creating default 'local' authorizer
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933151 25535 slave.cpp:208] Mesos agent started on (70)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933163 25535 slave.cpp:209] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_8cxdMK/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_8cxdMK/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_credentials="/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_8cxdMK/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_8cxdMK" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i"
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933385 25535 credentials.hpp:86] Loading credential for authentication from '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_8cxdMK/credential'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933447 25535 slave.cpp:346] Agent using credential for: test-principal
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933457 25535 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_8cxdMK/http_credentials'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933516 25535 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933557 25535 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933868 25535 slave.cpp:529] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933895 25535 slave.cpp:537] Agent attributes: [  ]
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933900 25535 slave.cpp:542] Agent hostname: ip-172-30-2-21.mesosphere.io
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.933909 25514 sched.cpp:226] Version: 1.1.0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934116 25534 sched.cpp:330] New master detected at master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934201 25534 sched.cpp:396] Authenticating with master master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934216 25534 sched.cpp:403] Using default CRAM-MD5 authenticatee
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934239 25535 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/meta'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934346 25533 authenticatee.cpp:121] Creating new client SASL connection
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934370 25536 status_update_manager.cpp:203] Recovering status update manager
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934443 25533 docker.cpp:745] Recovering Docker containers
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.934464 25533 docker.cpp:1093] Running docker -H unix:///var/run/docker.sock ps -a
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935253 25535 master.cpp:6234] Authenticating scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935492 25531 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(187)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935566 25531 authenticator.cpp:98] Creating new server SASL connection
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935650 25531 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935664 25531 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935730 25530 authenticator.cpp:204] Received SASL authentication start
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935782 25530 authenticator.cpp:326] Authentication requires more steps
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935829 25530 authenticatee.cpp:259] Received SASL authentication step
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935920 25529 authenticator.cpp:232] Received SASL authentication step
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935936 25529 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-21' server FQDN: 'ip-172-30-2-21' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935941 25529 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935950 25529 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935956 25529 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-21' server FQDN: 'ip-172-30-2-21' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935961 25529 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935964 25529 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.935972 25529 authenticator.cpp:318] Authentication success
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936008 25529 authenticatee.cpp:299] Authentication success
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936028 25530 master.cpp:6264] Successfully authenticated principal 'test-principal' at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936031 25529 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(187)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936146 25534 sched.cpp:502] Successfully authenticated with master master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936161 25534 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936252 25534 sched.cpp:853] Will retry registration in 1.798478411secs if necessary
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936300 25536 master.cpp:2429] Received SUBSCRIBE call for framework 'default' at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936324 25536 master.cpp:1891] Authorizing framework principal 'test-principal' to receive offers for role '*'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936439 25535 master.cpp:2505] Subscribing framework default with checkpointing disabled and capabilities [  ]
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936595 25530 hierarchical.cpp:269] Added framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936622 25530 hierarchical.cpp:1675] No allocations performed
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936633 25530 hierarchical.cpp:1770] No inverse offers to send out!
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936650 25530 hierarchical.cpp:1271] Performed allocation for 0 agents in 37890ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936700 25535 sched.cpp:743] Framework registered with 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.936722 25535 sched.cpp:757] Scheduler::registered took 9281ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977408 25532 slave.cpp:5197] Finished recovery
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977591 25532 slave.cpp:5369] Querying resource estimator for oversubscribable resources
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977715 25535 status_update_manager.cpp:177] Pausing sending status updates
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977741 25536 slave.cpp:911] New master detected at master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977759 25536 slave.cpp:970] Authenticating with master master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977777 25536 slave.cpp:981] Using default CRAM-MD5 authenticatee
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977810 25536 slave.cpp:943] Detecting new master
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977833 25533 authenticatee.cpp:121] Creating new client SASL connection
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977840 25536 slave.cpp:5383] Received oversubscribable resources  from the resource estimator
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977951 25536 master.cpp:6234] Authenticating slave(70)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.977995 25536 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(188)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978060 25536 authenticator.cpp:98] Creating new server SASL connection
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978119 25536 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978130 25536 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978160 25536 authenticator.cpp:204] Received SASL authentication start
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978196 25536 authenticator.cpp:326] Authentication requires more steps
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978227 25536 authenticatee.cpp:259] Received SASL authentication step
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978268 25536 authenticator.cpp:232] Received SASL authentication step
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978284 25536 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-21' server FQDN: 'ip-172-30-2-21' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978296 25536 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978307 25536 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978313 25536 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-21' server FQDN: 'ip-172-30-2-21' SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978318 25536 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978322 25536 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978329 25536 authenticator.cpp:318] Authentication success
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978373 25531 authenticatee.cpp:299] Authentication success
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978394 25530 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(188)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978421 25536 master.cpp:6264] Successfully authenticated principal 'test-principal' at slave(70)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978515 25535 slave.cpp:1065] Successfully authenticated with master master@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978570 25535 slave.cpp:1471] Will retry registration in 3.124457ms if necessary
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978649 25530 master.cpp:4886] Registering agent at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io) with id 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978767 25535 registrar.cpp:464] Applied 1 operations in 12383ns; attempting to update the registry
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.978977 25532 log.cpp:577] Attempting to append 390 bytes to the log
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.979030 25529 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.979323 25535 replica.cpp:537] Replica received write request for position 3 from __req_res__(872)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.981817 25529 slave.cpp:1471] Will retry registration in 11.003051ms if necessary
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.981868 25530 master.cpp:4874] Ignoring register agent message from slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io) as admission is already in progress
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.982168 25535 leveldb.cpp:341] Persisting action (409 bytes) to leveldb took 2.827325ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.982183 25535 replica.cpp:708] Persisted action APPEND at position 3
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.982389 25534 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985191 25534 leveldb.cpp:341] Persisting action (411 bytes) to leveldb took 2.786436ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985208 25534 replica.cpp:708] Persisted action APPEND at position 3
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985467 25533 registrar.cpp:509] Successfully updated the registry in 6.67904ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985527 25531 log.cpp:596] Attempting to truncate the log to 3
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985571 25532 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985651 25530 master.cpp:4956] Registered agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985708 25532 slave.cpp:4064] Received ping from slave-observer(67)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985800 25536 hierarchical.cpp:476] Added agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 (ip-172-30-2-21.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: )
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985854 25532 slave.cpp:1111] Registered with master master@172.30.2.21:53211; given agent ID 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985867 25532 fetcher.cpp:86] Clearing fetcher cache
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985868 25529 replica.cpp:537] Replica received write request for position 4 from __req_res__(873)@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.985935 25533 status_update_manager.cpp:184] Resuming sending status updates
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986016 25536 hierarchical.cpp:1770] No inverse offers to send out!
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986026 25532 slave.cpp:1134] Checkpointing SlaveInfo to '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/meta/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/slave.info'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986032 25536 hierarchical.cpp:1294] Performed allocation for agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 in 206746ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986106 25530 master.cpp:6063] Sending 1 offers to framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986122 25532 slave.cpp:1171] Forwarding total oversubscribed resources 
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986174 25530 master.cpp:5340] Received update of agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io) with total oversubscribed resources 
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986246 25530 hierarchical.cpp:540] Agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 (ip-172-30-2-21.mesosphere.io) updated with oversubscribed resources  (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000])
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986279 25530 hierarchical.cpp:1675] No allocations performed
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986289 25530 hierarchical.cpp:1770] No inverse offers to send out!
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986290 25532 sched.cpp:917] Scheduler::resourceOffers took 41755ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986302 25530 hierarchical.cpp:1294] Performed allocation for agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 in 35768ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986865 25531 master.cpp:3363] Processing ACCEPT call for offers: [ 4227c978-ffc0-453f-b611-353e1665bd0f-O0 ] on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io) for framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.986893 25531 master.cpp:2985] Authorizing framework principal 'test-principal' to launch task 1
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.987294 25535 master.cpp:7809] Adding task 1 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 (ip-172-30-2-21.mesosphere.io)
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.987341 25535 master.cpp:3963] Launching task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.987540 25533 slave.cpp:1535] Got assigned task '1' for framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.987725 25533 slave.cpp:1692] Launching task '1' for framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.988034 25533 paths.cpp:536] Trying to chown '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000/executors/1/runs/ef09d876-0b1d-4830-b1b0-38b642ac5240' to user 'root'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.988934 25529 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 3.009008ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.988961 25529 replica.cpp:708] Persisted action TRUNCATE at position 4
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.989442 25534 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.991277 25533 slave.cpp:6089] Launching executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000/executors/1/runs/ef09d876-0b1d-4830-b1b0-38b642ac5240'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.991497 25533 slave.cpp:1978] Queued task '1' for executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.991539 25533 slave.cpp:864] Successfully attached file '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000/executors/1/runs/ef09d876-0b1d-4830-b1b0-38b642ac5240'
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.992271 25534 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 2.788408ms
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.992318 25534 leveldb.cpp:399] Deleting ~2 keys from leveldb took 20355ns
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.992328 25534 replica.cpp:708] Persisted action TRUNCATE at position 4
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.994469 25535 docker.cpp:1014] Starting container 'ef09d876-0b1d-4830-b1b0-38b642ac5240' for task '1' (and executor '1') of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:48]W:	 [Step 10/10] I0915 03:00:48.994659 25535 docker.cpp:1261] Running docker -H unix:///var/run/docker.sock inspect alpine:latest
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.078346 25530 docker.cpp:422] Docker pull alpine completed
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.078583 25530 docker.cpp:1322] Launching 'mesos-docker-executor' with flags '--container="mesos-4227c978-ffc0-453f-b611-353e1665bd0f-S0.ef09d876-0b1d-4830-b1b0-38b642ac5240" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000/executors/1/runs/ef09d876-0b1d-4830-b1b0-38b642ac5240" --stop_timeout="0ns"'
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.112851 25532 slave.cpp:3178] Got registration for executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.113209 25536 docker.cpp:1417] Ignoring updating container 'ef09d876-0b1d-4830-b1b0-38b642ac5240' with resources passed to update is identical to existing resources
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.113327 25532 slave.cpp:2182] Sending queued task '1' to executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 at executor(1)@172.30.2.21:53822
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.821586 25535 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.821759 25535 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.821779 25535 status_update_manager.cpp:500] Creating StatusUpdate stream for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.821964 25535 status_update_manager.cpp:377] Forwarding update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to the agent
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822052 25533 slave.cpp:3982] Forwarding the update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to master@172.30.2.21:53211
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822142 25533 slave.cpp:3876] Status update manager successfully handled status update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822163 25533 slave.cpp:3892] Sending acknowledgement for status update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to executor(1)@172.30.2.21:53822
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822268 25530 master.cpp:5475] Status update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822298 25530 master.cpp:5537] Forwarding status update TASK_RUNNING (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822362 25530 master.cpp:7187] Updating the state of task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822548 25535 sched.cpp:1025] Scheduler::statusUpdate took 43132ns
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822773 25532 master.cpp:4598] Processing ACKNOWLEDGE call bc084c8d-21eb-4071-b48a-5a07618b1f07 for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211 on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822890 25535 status_update_manager.cpp:395] Received status update acknowledgement (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.822952 25535 slave.cpp:2928] Status update manager successfully handled status update acknowledgement (UUID: bc084c8d-21eb-4071-b48a-5a07618b1f07) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.890867 25535 hierarchical.cpp:1675] No allocations performed
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.890877 25535 hierarchical.cpp:1770] No inverse offers to send out!
[03:00:49]W:	 [Step 10/10] I0915 03:00:49.890897 25535 hierarchical.cpp:1271] Performed allocation for 1 agents in 55218ns
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023056 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023205 25533 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023272 25533 status_update_manager.cpp:377] Forwarding update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to the agent
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023383 25533 slave.cpp:3982] Forwarding the update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to master@172.30.2.21:53211
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023434 25533 slave.cpp:3876] Status update manager successfully handled status update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023447 25533 slave.cpp:3892] Sending acknowledgement for status update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023563 25533 master.cpp:5475] Status update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023582 25533 master.cpp:5537] Forwarding status update TASK_RUNNING (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023617 25533 master.cpp:7187] Updating the state of task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023820 25533 sched.cpp:1025] Scheduler::statusUpdate took 40443ns
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.023970 25534 master.cpp:4598] Processing ACKNOWLEDGE call 0fd84561-56bd-4d33-97e9-1ca1214f22ee for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211 on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.024096 25531 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.024168 25531 slave.cpp:2928] Status update manager successfully handled status update acknowledgement (UUID: 0fd84561-56bd-4d33-97e9-1ca1214f22ee) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123150 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123291 25529 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123348 25529 status_update_manager.cpp:377] Forwarding update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to the agent
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123441 25530 slave.cpp:3982] Forwarding the update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to master@172.30.2.21:53211
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123530 25530 slave.cpp:3876] Status update manager successfully handled status update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123553 25530 slave.cpp:3892] Sending acknowledgement for status update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123584 25529 master.cpp:5475] Status update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123602 25529 master.cpp:5537] Forwarding status update TASK_RUNNING (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123661 25529 master.cpp:7187] Updating the state of task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123817 25534 sched.cpp:1025] Scheduler::statusUpdate took 34747ns
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.123988 25536 master.cpp:4598] Processing ACKNOWLEDGE call 19a8266e-3653-4d6a-a581-a6726b14134b for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211 on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.124095 25531 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.124184 25531 slave.cpp:2928] Status update manager successfully handled status update acknowledgement (UUID: 19a8266e-3653-4d6a-a581-a6726b14134b) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224483 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224619 25535 status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224674 25535 status_update_manager.cpp:377] Forwarding update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to the agent
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224759 25534 slave.cpp:3982] Forwarding the update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to master@172.30.2.21:53211
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224817 25534 slave.cpp:3876] Status update manager successfully handled status update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224831 25534 slave.cpp:3892] Sending acknowledgement for status update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 to executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224916 25531 master.cpp:5475] Status update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224937 25531 master.cpp:5537] Forwarding status update TASK_RUNNING (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.224982 25531 master.cpp:7187] Updating the state of task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225129 25534 sched.cpp:1025] Scheduler::statusUpdate took 31352ns
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225247 25534 master.cpp:4598] Processing ACKNOWLEDGE call 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214 for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211 on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225339 25534 status_update_manager.cpp:395] Received status update acknowledgement (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225373 25514 sched.cpp:1987] Asked to stop the driver
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225419 25529 slave.cpp:2928] Status update manager successfully handled status update acknowledgement (UUID: 027d2dc4-e8cc-4ffc-a2c8-b9242f68a214) for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225435 25535 sched.cpp:1187] Stopping framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225551 25531 master.cpp:6639] Processing TEARDOWN call for framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225566 25531 master.cpp:6651] Removing framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (default) at scheduler-9a87d6da-4e45-4e0b-b0d9-b8f339ed78e5@172.30.2.21:53211
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225641 25536 hierarchical.cpp:380] Deactivated framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225663 25531 master.cpp:7187] Updating the state of task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225670 25530 slave.cpp:2496] Asked to shut down framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 by master@172.30.2.21:53211
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225687 25530 slave.cpp:2521] Shutting down framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225698 25530 slave.cpp:4785] Shutting down executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 at executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225822 25531 master.cpp:7283] Removing task 1 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225860 25536 hierarchical.cpp:1003] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 from framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.225999 25531 hierarchical.cpp:331] Removed framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.324059 25532 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 201ef2e8-0637-4dcc-a02e-a7f7b5dd7f9b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.324090 25532 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 201ef2e8-0637-4dcc-a02e-a7f7b5dd7f9b) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.424906 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: d8d3768f-7531-4800-ad90-b96edb61ac38) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.424932 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: d8d3768f-7531-4800-ad90-b96edb61ac38) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.525677 25533 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: ba49b4f2-9834-4d17-8172-3e5d350420b1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.525702 25533 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: ba49b4f2-9834-4d17-8172-3e5d350420b1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.626348 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: a944b1ad-e146-4191-b74e-2d1214d69998) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.626368 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: a944b1ad-e146-4191-b74e-2d1214d69998) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.727735 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 94ee7b6c-25d9-4650-b5f2-e66376d8b01a) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.727763 25531 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 94ee7b6c-25d9-4650-b5f2-e66376d8b01a) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.827417 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 3ab8c9cd-8f21-4c42-8cce-7b1f882602a1) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.827440 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 3ab8c9cd-8f21-4c42-8cce-7b1f882602a1) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.891840 25532 hierarchical.cpp:1675] No allocations performed
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.891866 25532 hierarchical.cpp:1271] Performed allocation for 1 agents in 74869ns
[03:00:50]W:	 [Step 10/10] I0915 03:00:50.928840 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: bb1a9315-b1f9-4264-942a-1b681d5824db) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:50]W:	 [Step 10/10] W0915 03:00:50.928861 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: bb1a9315-b1f9-4264-942a-1b681d5824db) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.028501 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 0b10a8a9-6339-4bcf-a32b-062de5a31263) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.028523 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 0b10a8a9-6339-4bcf-a32b-062de5a31263) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.129211 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 0be14ca4-b5f6-4def-baef-406c06543536) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.129240 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 0be14ca4-b5f6-4def-baef-406c06543536) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.229909 25532 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 00d45f6f-74ba-44df-bbff-174780f1459d) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.229930 25532 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 00d45f6f-74ba-44df-bbff-174780f1459d) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.330664 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 8f6d78ea-8a3c-446e-8aaa-e86caad10739) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.330687 25531 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 8f6d78ea-8a3c-446e-8aaa-e86caad10739) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.431432 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 3d8260c2-8ceb-4d6c-aa2e-e6abe0468e97) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.431452 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 3d8260c2-8ceb-4d6c-aa2e-e6abe0468e97) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.532104 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 931ac2af-1032-401d-b5f0-9003e5c85e27) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.532125 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 931ac2af-1032-401d-b5f0-9003e5c85e27) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.632899 25533 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 5b5e44a6-043f-4848-8fbb-718510e14af2) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.632920 25533 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 5b5e44a6-043f-4848-8fbb-718510e14af2) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.733582 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: b9cbbf1a-0f5e-4490-9e23-c4c9cb9d15b5) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.733603 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: b9cbbf1a-0f5e-4490-9e23-c4c9cb9d15b5) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.834338 25532 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 1402e577-3bf7-4f23-8332-114b6b2d5c6f) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.834367 25532 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 1402e577-3bf7-4f23-8332-114b6b2d5c6f) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.892592 25535 hierarchical.cpp:1675] No allocations performed
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.892623 25535 hierarchical.cpp:1271] Performed allocation for 1 agents in 80902ns
[03:00:51]W:	 [Step 10/10] I0915 03:00:51.935051 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 9ff10665-5314-4c89-85cb-476679085859) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:51]W:	 [Step 10/10] W0915 03:00:51.935075 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 9ff10665-5314-4c89-85cb-476679085859) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.035720 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: f6a68e93-346e-4be6-a606-7b252e77c56a) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.035742 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: f6a68e93-346e-4be6-a606-7b252e77c56a) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.136411 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: e38d5ac2-7c7e-4f9a-bccb-88146ae2ecd6) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.136431 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: e38d5ac2-7c7e-4f9a-bccb-88146ae2ecd6) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.237196 25533 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: a3aaf713-e57a-4f0c-ab65-26b14a7b4382) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.237217 25533 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: a3aaf713-e57a-4f0c-ab65-26b14a7b4382) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.337995 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 313e6d2c-7ce1-403e-9585-c07f6f3ada22) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.338027 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 313e6d2c-7ce1-403e-9585-c07f6f3ada22) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.438653 25532 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 10263dc9-0522-4734-9265-d4cc3f7a7e14) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.438674 25532 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 10263dc9-0522-4734-9265-d4cc3f7a7e14) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.539347 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: fd0ddb01-2391-4376-b47f-bc1b61eaa4d1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.539371 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: fd0ddb01-2391-4376-b47f-bc1b61eaa4d1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.640055 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 32d4e984-bc83-45e5-812a-885ac49cdcba) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.640079 25531 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 32d4e984-bc83-45e5-812a-885ac49cdcba) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.740753 25535 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: efdd0bdb-eed3-4a66-9305-dedcf558dcee) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.740774 25535 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: efdd0bdb-eed3-4a66-9305-dedcf558dcee) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.842142 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 4eeacea3-15fd-4725-ae84-4d031cf59496) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.842162 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 4eeacea3-15fd-4725-ae84-4d031cf59496) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.893647 25530 hierarchical.cpp:1675] No allocations performed
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.893678 25530 hierarchical.cpp:1271] Performed allocation for 1 agents in 78701ns
[03:00:52]W:	 [Step 10/10] I0915 03:00:52.941725 25533 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 2fc8bdb3-a5f4-4751-a41f-047ee9d4aadc) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:52]W:	 [Step 10/10] W0915 03:00:52.941745 25533 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 2fc8bdb3-a5f4-4751-a41f-047ee9d4aadc) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.042394 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 5a920458-aeb6-43ea-9d47-0dd9372c0423) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.042415 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 5a920458-aeb6-43ea-9d47-0dd9372c0423) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.143177 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 751b4ca2-dfab-4bd1-9dba-f1fefae6ef9c) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.143199 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 751b4ca2-dfab-4bd1-9dba-f1fefae6ef9c) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.243940 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 5bea45b3-b83b-4b69-96b8-321890f91a6b) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.243965 25531 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 5bea45b3-b83b-4b69-96b8-321890f91a6b) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.344749 25532 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 2db0dc6d-5e55-4232-9b6e-255fb2d1d8e1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.344775 25532 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 2db0dc6d-5e55-4232-9b6e-255fb2d1d8e1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.445394 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 8e2694a9-ef45-4077-bd27-2cadfd046422) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.445417 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 8e2694a9-ef45-4077-bd27-2cadfd046422) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.546133 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 041862a9-15e6-4d11-8ddf-c071db11fbdd) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.546165 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 041862a9-15e6-4d11-8ddf-c071db11fbdd) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.646854 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: ac5484a5-801a-442c-9755-2674f21ebfba) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.646877 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: ac5484a5-801a-442c-9755-2674f21ebfba) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.747555 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 486f8cd5-36e7-4b28-af57-ad50affe7a89) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.747575 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 486f8cd5-36e7-4b28-af57-ad50affe7a89) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.848328 25535 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 81bb6e8a-efed-4673-97e7-f8fb0931bb49) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.848352 25535 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 81bb6e8a-efed-4673-97e7-f8fb0931bb49) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.894559 25531 hierarchical.cpp:1675] No allocations performed
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.894583 25531 hierarchical.cpp:1271] Performed allocation for 1 agents in 62503ns
[03:00:53]W:	 [Step 10/10] I0915 03:00:53.949090 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: cbfbf344-f1f5-4da0-91f6-1ac9852c8106) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:53]W:	 [Step 10/10] W0915 03:00:53.949111 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: cbfbf344-f1f5-4da0-91f6-1ac9852c8106) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.049875 25533 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 6c51606b-ca6e-4add-a2e8-a3e49704c2eb) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.049899 25533 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 6c51606b-ca6e-4add-a2e8-a3e49704c2eb) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.150591 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 38a10a19-4b8e-4a3b-a783-cba82bde4ace) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.150614 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 38a10a19-4b8e-4a3b-a783-cba82bde4ace) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.251353 25534 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: b3af0e36-1a25-4d1e-a86b-24cec3d2cfae) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.251374 25534 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: b3af0e36-1a25-4d1e-a86b-24cec3d2cfae) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.352131 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 56ab0cf3-6290-4e75-92f5-638db392c84a) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.352159 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 56ab0cf3-6290-4e75-92f5-638db392c84a) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.452885 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: c5bdd0af-76f7-4913-92a6-5a727d913565) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.452914 25531 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: c5bdd0af-76f7-4913-92a6-5a727d913565) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.553601 25535 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: b4fc33e0-7431-46dc-a06c-d9965c860463) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.553627 25535 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: b4fc33e0-7431-46dc-a06c-d9965c860463) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.654302 25532 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 6ac88684-cc48-4c50-bbfd-90357bc053c5) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.654325 25532 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 6ac88684-cc48-4c50-bbfd-90357bc053c5) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.755060 25536 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: a9c0ad82-b4c2-4636-a30d-c84726dcfdcb) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.755092 25536 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: a9c0ad82-b4c2-4636-a30d-c84726dcfdcb) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.856001 25533 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 634e84e9-04d4-48b0-a888-fd76108f675c) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.856025 25533 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 634e84e9-04d4-48b0-a888-fd76108f675c) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.895265 25534 hierarchical.cpp:1675] No allocations performed
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.895292 25534 hierarchical.cpp:1271] Performed allocation for 1 agents in 72993ns
[03:00:54]W:	 [Step 10/10] I0915 03:00:54.956718 25529 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 597e0d0f-ebf5-4e3d-9b4d-1f6db6215ff1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:54]W:	 [Step 10/10] W0915 03:00:54.956743 25529 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 597e0d0f-ebf5-4e3d-9b4d-1f6db6215ff1) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.057342 25530 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 4b6b444e-fe36-4026-b28c-a9e69b1d7a05) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:55]W:	 [Step 10/10] W0915 03:00:55.057370 25530 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 4b6b444e-fe36-4026-b28c-a9e69b1d7a05) for task 1 in health state unhealthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.158027 25531 slave.cpp:3581] Handling status update TASK_RUNNING (UUID: 45fb6b01-7aeb-4e1c-b104-2863f40a28bb) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 from executor(1)@172.30.2.21:53822
[03:00:55]W:	 [Step 10/10] W0915 03:00:55.158046 25531 slave.cpp:3634] Ignoring status update TASK_RUNNING (UUID: 45fb6b01-7aeb-4e1c-b104-2863f40a28bb) for task 1 in health state healthy of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 for terminating framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.226503 25536 slave.cpp:4858] Killing executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 at executor(1)@172.30.2.21:53822
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.226559 25536 docker.cpp:1832] Destroying container 'ef09d876-0b1d-4830-b1b0-38b642ac5240'
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.226568 25536 docker.cpp:1918] Sending SIGTERM to executor with pid: 316
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.228757 25530 slave.cpp:4110] Got exited event for executor(1)@172.30.2.21:53822
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.242672 25536 docker.cpp:1960] Running docker stop on container 'ef09d876-0b1d-4830-b1b0-38b642ac5240'
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.242722 25536 docker.cpp:862] Running docker -H unix:///var/run/docker.sock stop -t 0 mesos-4227c978-ffc0-453f-b611-353e1665bd0f-S0.ef09d876-0b1d-4830-b1b0-38b642ac5240
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318176 25534 docker.cpp:2112] Executor for container 'ef09d876-0b1d-4830-b1b0-38b642ac5240' has exited
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318680 25529 slave.cpp:4467] Executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 terminated with signal Killed
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318702 25529 slave.cpp:4571] Cleaning up executor '1' of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000 at executor(1)@172.30.2.21:53822
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318843 25534 gc.cpp:55] Scheduling '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000/executors/1/runs/ef09d876-0b1d-4830-b1b0-38b642ac5240' for gc 6.99999631038222days in the future
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318876 25529 slave.cpp:4659] Cleaning up framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318892 25534 gc.cpp:55] Scheduling '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000/executors/1' for gc 6.99999630943111days in the future
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318915 25534 status_update_manager.cpp:285] Closing status update streams for framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318925 25534 status_update_manager.cpp:531] Cleaning up status update stream for task 1 of framework 4227c978-ffc0-453f-b611-353e1665bd0f-0000
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.318995 25529 slave.cpp:783] Agent terminating
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.319012 25535 gc.cpp:55] Scheduling '/mnt/teamcity/temp/buildTmp/HealthCheckTest_ROOT_DOCKER_DockerHealthStatusChange_t2tx3i/slaves/4227c978-ffc0-453f-b611-353e1665bd0f-S0/frameworks/4227c978-ffc0-453f-b611-353e1665bd0f-0000' for gc 6.99999630869333days in the future
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.319092 25529 master.cpp:1254] Agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io) disconnected
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.319111 25529 master.cpp:2789] Disconnecting agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.319147 25529 master.cpp:2808] Deactivating agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 at slave(70)@172.30.2.21:53211 (ip-172-30-2-21.mesosphere.io)
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.319212 25532 hierarchical.cpp:569] Agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0 deactivated
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.320049 25514 docker.cpp:1093] Running docker -H unix:///var/run/docker.sock ps -a
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.418561 25532 docker.cpp:974] Running docker -H unix:///var/run/docker.sock inspect mesos-4227c978-ffc0-453f-b611-353e1665bd0f-S0.ef09d876-0b1d-4830-b1b0-38b642ac5240
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.521036 25534 docker.cpp:334] Unable to detect IP Address at 'NetworkSettings.Networks.host.IPAddress', attempting deprecated field
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.521400 25514 docker.cpp:935] Running docker -H unix:///var/run/docker.sock rm -f -v 58baa0ece7643eaaae67a83c3682b65d20d75376e11070dd0eb80536d19bfbda
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.896472 25530 hierarchical.cpp:1675] No allocations performed
[03:00:55]W:	 [Step 10/10] I0915 03:00:55.896502 25530 hierarchical.cpp:1271] Performed allocation for 1 agents in 66305ns
[03:00:56]W:	 [Step 10/10] I0915 03:00:56.897131 25536 hierarchical.cpp:1675] No allocations performed
[03:00:56]W:	 [Step 10/10] I0915 03:00:56.897163 25536 hierarchical.cpp:1271] Performed allocation for 1 agents in 80640ns
[03:00:57] :	 [Step 10/10] ../../src/tests/health_check_tests.cpp:1024: Failure
[03:00:57] :	 [Step 10/10] (docker->rm(container.id, true)).failure(): Failed to run 'docker -H unix:///var/run/docker.sock rm -f -v 58baa0ece7643eaaae67a83c3682b65d20d75376e11070dd0eb80536d19bfbda': exited with status 1; stderr='Error response from daemon: Cannot destroy container 58baa0ece7643eaaae67a83c3682b65d20d75376e11070dd0eb80536d19bfbda: Could not kill running container, cannot remove - [2] Container does not exist: container destroyed
[03:00:57] :	 [Step 10/10] Error: failed to remove containers: [58baa0ece7643eaaae67a83c3682b65d20d75376e11070dd0eb80536d19bfbda]
[03:00:57] :	 [Step 10/10] '
[03:00:57]W:	 [Step 10/10] I0915 03:00:57.630208 25514 master.cpp:1097] Master terminating
[03:00:57]W:	 [Step 10/10] I0915 03:00:57.630409 25531 hierarchical.cpp:508] Removed agent 4227c978-ffc0-453f-b611-353e1665bd0f-S0
[03:00:57] :	 [Step 10/10] [  FAILED  ] HealthCheckTest.ROOT_DOCKER_DockerHealthStatusChange (8839 ms)

Attachments

Issue Links

relates to

MESOS-1802 HealthCheckTest.HealthStatusChange is flaky on jenkins.

Resolved

Activity

People

Assignee:: Unassigned

Reporter:: Greg Mann

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 16/Sep/16 00:57

Updated:: 06/Feb/19 17:29