Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
1.0.0
-
None
-
Ubuntu 12
Description
Looks like there is still some flakiness left from MESOS-2858 that was fixed recently.
Showed up on the internal CI:
[17:58:34] : [Step 10/10] [ RUN ] FetcherCacheHttpTest.HttpMixed [17:58:34]W: [Step 10/10] I0324 17:58:34.392750 25227 cluster.cpp:139] Creating default 'local' authorizer [17:58:34]W: [Step 10/10] I0324 17:58:34.406461 25227 leveldb.cpp:174] Opened db in 13.518176ms [17:58:34]W: [Step 10/10] I0324 17:58:34.408825 25227 leveldb.cpp:181] Compacted db in 2.338447ms [17:58:34]W: [Step 10/10] I0324 17:58:34.408866 25227 leveldb.cpp:196] Created db iterator in 16930ns [17:58:34]W: [Step 10/10] I0324 17:58:34.408879 25227 leveldb.cpp:202] Seeked to beginning of db in 1521ns [17:58:34]W: [Step 10/10] I0324 17:58:34.408885 25227 leveldb.cpp:271] Iterated through 0 keys in the db in 312ns [17:58:34]W: [Step 10/10] I0324 17:58:34.408921 25227 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned [17:58:34]W: [Step 10/10] I0324 17:58:34.409371 25241 recover.cpp:447] Starting replica recovery [17:58:34]W: [Step 10/10] I0324 17:58:34.409600 25241 recover.cpp:473] Replica is in EMPTY status [17:58:34]W: [Step 10/10] I0324 17:58:34.410581 25248 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (2680)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.410959 25243 recover.cpp:193] Received a recover response from a replica in EMPTY status [17:58:34]W: [Step 10/10] I0324 17:58:34.411314 25248 recover.cpp:564] Updating replica status to STARTING [17:58:34]W: [Step 10/10] I0324 17:58:34.412307 25242 master.cpp:376] Master 9a1809d1-13d2-41fb-bc50-5fcd03a1082d (ip-172-30-2-190.mesosphere.io) started on 172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.412328 25242 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/B9KLC9/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/B9KLC9/master" --zk_session_timeout="10secs" [17:58:34]W: [Step 10/10] I0324 17:58:34.412554 25242 master.cpp:427] Master only allowing authenticated frameworks to register [17:58:34]W: [Step 10/10] I0324 17:58:34.412562 25242 master.cpp:432] Master only allowing authenticated slaves to register [17:58:34]W: [Step 10/10] I0324 17:58:34.412569 25242 credentials.hpp:35] Loading credentials for authentication from '/tmp/B9KLC9/credentials' [17:58:34]W: [Step 10/10] I0324 17:58:34.413414 25242 master.cpp:474] Using default 'crammd5' authenticator [17:58:34]W: [Step 10/10] I0324 17:58:34.413554 25242 master.cpp:545] Using default 'basic' HTTP authenticator [17:58:34]W: [Step 10/10] I0324 17:58:34.413723 25242 master.cpp:583] Authorization enabled [17:58:34]W: [Step 10/10] I0324 17:58:34.413754 25245 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 2.257079ms [17:58:34]W: [Step 10/10] I0324 17:58:34.413781 25245 replica.cpp:320] Persisted replica status to STARTING [17:58:34]W: [Step 10/10] I0324 17:58:34.413907 25247 hierarchical.cpp:144] Initialized hierarchical allocator process [17:58:34]W: [Step 10/10] I0324 17:58:34.413954 25241 whitelist_watcher.cpp:77] No whitelist given [17:58:34]W: [Step 10/10] I0324 17:58:34.414001 25244 recover.cpp:473] Replica is in STARTING status [17:58:34]W: [Step 10/10] I0324 17:58:34.414772 25248 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (2682)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.415112 25246 recover.cpp:193] Received a recover response from a replica in STARTING status [17:58:34]W: [Step 10/10] I0324 17:58:34.415455 25241 recover.cpp:564] Updating replica status to VOTING [17:58:34]W: [Step 10/10] I0324 17:58:34.416200 25245 master.cpp:1826] The newly elected leader is master@172.30.2.190:33501 with id 9a1809d1-13d2-41fb-bc50-5fcd03a1082d [17:58:34]W: [Step 10/10] I0324 17:58:34.416224 25245 master.cpp:1839] Elected as the leading master! [17:58:34]W: [Step 10/10] I0324 17:58:34.416235 25245 master.cpp:1526] Recovering from registrar [17:58:34]W: [Step 10/10] I0324 17:58:34.416363 25243 registrar.cpp:307] Recovering registrar [17:58:34]W: [Step 10/10] I0324 17:58:34.417546 25248 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.934949ms [17:58:34]W: [Step 10/10] I0324 17:58:34.417567 25248 replica.cpp:320] Persisted replica status to VOTING [17:58:34]W: [Step 10/10] I0324 17:58:34.417667 25245 recover.cpp:578] Successfully joined the Paxos group [17:58:34]W: [Step 10/10] I0324 17:58:34.417815 25245 recover.cpp:462] Recover process terminated [17:58:34]W: [Step 10/10] I0324 17:58:34.418218 25244 log.cpp:659] Attempting to start the writer [17:58:34]W: [Step 10/10] I0324 17:58:34.419313 25247 replica.cpp:493] Replica received implicit promise request from (2683)@172.30.2.190:33501 with proposal 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.421195 25247 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.851938ms [17:58:34]W: [Step 10/10] I0324 17:58:34.421216 25247 replica.cpp:342] Persisted promised to 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.421702 25246 coordinator.cpp:238] Coordinator attempting to fill missing positions [17:58:34]W: [Step 10/10] I0324 17:58:34.422874 25242 replica.cpp:388] Replica received explicit promise request from (2684)@172.30.2.190:33501 for position 0 with proposal 2 [17:58:34]W: [Step 10/10] I0324 17:58:34.423462 25242 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 552744ns [17:58:34]W: [Step 10/10] I0324 17:58:34.423481 25242 replica.cpp:712] Persisted action at 0 [17:58:34]W: [Step 10/10] I0324 17:58:34.424360 25243 replica.cpp:537] Replica received write request for position 0 from (2685)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.424408 25243 leveldb.cpp:436] Reading position from leveldb took 23978ns [17:58:34]W: [Step 10/10] I0324 17:58:34.426209 25243 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 1.769639ms [17:58:34]W: [Step 10/10] I0324 17:58:34.426231 25243 replica.cpp:712] Persisted action at 0 [17:58:34]W: [Step 10/10] I0324 17:58:34.426774 25245 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 [17:58:34]W: [Step 10/10] I0324 17:58:34.427505 25245 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 705580ns [17:58:34]W: [Step 10/10] I0324 17:58:34.427523 25245 replica.cpp:712] Persisted action at 0 [17:58:34]W: [Step 10/10] I0324 17:58:34.427537 25245 replica.cpp:697] Replica learned NOP action at position 0 [17:58:34]W: [Step 10/10] I0324 17:58:34.428007 25245 log.cpp:675] Writer started with ending position 0 [17:58:34]W: [Step 10/10] I0324 17:58:34.428905 25246 leveldb.cpp:436] Reading position from leveldb took 28609ns [17:58:34]W: [Step 10/10] I0324 17:58:34.429667 25241 registrar.cpp:340] Successfully fetched the registry (0B) in 13.2672ms [17:58:34]W: [Step 10/10] I0324 17:58:34.429766 25241 registrar.cpp:439] Applied 1 operations in 22565ns; attempting to update the 'registry' [17:58:34]W: [Step 10/10] I0324 17:58:34.430347 25243 log.cpp:683] Attempting to append 210 bytes to the log [17:58:34]W: [Step 10/10] I0324 17:58:34.430445 25244 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.431056 25248 replica.cpp:537] Replica received write request for position 1 from (2686)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.433200 25248 leveldb.cpp:341] Persisting action (229 bytes) to leveldb took 2.107857ms [17:58:34]W: [Step 10/10] I0324 17:58:34.433223 25248 replica.cpp:712] Persisted action at 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.433760 25245 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 [17:58:34]W: [Step 10/10] I0324 17:58:34.435839 25245 leveldb.cpp:341] Persisting action (231 bytes) to leveldb took 2.05152ms [17:58:34]W: [Step 10/10] I0324 17:58:34.435863 25245 replica.cpp:712] Persisted action at 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.435878 25245 replica.cpp:697] Replica learned APPEND action at position 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.436672 25244 registrar.cpp:484] Successfully updated the 'registry' in 6.852864ms [17:58:34]W: [Step 10/10] I0324 17:58:34.436782 25244 registrar.cpp:370] Successfully recovered registrar [17:58:34]W: [Step 10/10] I0324 17:58:34.436919 25245 log.cpp:702] Attempting to truncate the log to 1 [17:58:34]W: [Step 10/10] I0324 17:58:34.437021 25248 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 [17:58:34]W: [Step 10/10] I0324 17:58:34.437201 25242 master.cpp:1634] Recovered 0 slaves from the Registry (171B) ; allowing 10mins for slaves to re-register [17:58:34]W: [Step 10/10] I0324 17:58:34.437218 25245 hierarchical.cpp:171] Skipping recovery of hierarchical allocator: nothing to recover [17:58:34]W: [Step 10/10] I0324 17:58:34.437778 25242 replica.cpp:537] Replica received write request for position 2 from (2687)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.508529 25242 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 70.721239ms [17:58:34]W: [Step 10/10] I0324 17:58:34.508554 25242 replica.cpp:712] Persisted action at 2 [17:58:34]W: [Step 10/10] I0324 17:58:34.509089 25242 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 [17:58:34]W: [Step 10/10] I0324 17:58:34.510949 25242 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 1.837944ms [17:58:34]W: [Step 10/10] I0324 17:58:34.510994 25242 leveldb.cpp:399] Deleting ~1 keys from leveldb took 23052ns [17:58:34]W: [Step 10/10] I0324 17:58:34.511008 25242 replica.cpp:712] Persisted action at 2 [17:58:34]W: [Step 10/10] I0324 17:58:34.511023 25242 replica.cpp:697] Replica learned TRUNCATE action at position 2 [17:58:34]W: [Step 10/10] I0324 17:58:34.519076 25227 containerizer.cpp:149] Using isolation: posix/cpu,posix/mem,filesystem/posix [17:58:34]W: [Step 10/10] I0324 17:58:34.523919 25227 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher [17:58:34]W: [Step 10/10] I0324 17:58:34.527410 25244 slave.cpp:200] Slave started on 57)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.527428 25244 slave.cpp:201] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="true" --hostname_lookup="true" --http_authenticators="basic" --http_credentials="/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:1000;mem:1000" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9" [17:58:34]W: [Step 10/10] I0324 17:58:34.527875 25244 credentials.hpp:83] Loading credential for authentication from '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/credential' [17:58:34]W: [Step 10/10] I0324 17:58:34.528064 25244 slave.cpp:338] Slave using credential for: test-principal [17:58:34]W: [Step 10/10] I0324 17:58:34.528089 25244 credentials.hpp:35] Loading credentials for authentication from '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/http_credentials' [17:58:34]W: [Step 10/10] I0324 17:58:34.528254 25244 slave.cpp:390] Using default 'basic' HTTP authenticator [17:58:34]W: [Step 10/10] I0324 17:58:34.528488 25244 resources.cpp:572] Parsing resources as JSON failed: cpus:1000;mem:1000 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.528828 25244 slave.cpp:589] Slave resources: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] [17:58:34]W: [Step 10/10] I0324 17:58:34.528884 25244 slave.cpp:597] Slave attributes: [ ] [17:58:34]W: [Step 10/10] I0324 17:58:34.528894 25244 slave.cpp:602] Slave hostname: ip-172-30-2-190.mesosphere.io [17:58:34]W: [Step 10/10] I0324 17:58:34.529839 25247 state.cpp:58] Recovering state from '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta' [17:58:34]W: [Step 10/10] I0324 17:58:34.530105 25243 status_update_manager.cpp:200] Recovering status update manager [17:58:34]W: [Step 10/10] I0324 17:58:34.530272 25247 containerizer.cpp:407] Recovering containerizer [17:58:34]W: [Step 10/10] I0324 17:58:34.531729 25243 provisioner.cpp:245] Provisioner recovery complete [17:58:34]W: [Step 10/10] I0324 17:58:34.532035 25244 slave.cpp:4726] Finished recovery [17:58:34]W: [Step 10/10] I0324 17:58:34.532284 25244 slave.cpp:4898] Querying resource estimator for oversubscribable resources [17:58:34]W: [Step 10/10] I0324 17:58:34.532476 25246 status_update_manager.cpp:174] Pausing sending status updates [17:58:34]W: [Step 10/10] I0324 17:58:34.532491 25244 slave.cpp:928] New master detected at master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.532551 25244 slave.cpp:991] Authenticating with master master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.532564 25244 slave.cpp:996] Using default CRAM-MD5 authenticatee [17:58:34]W: [Step 10/10] I0324 17:58:34.532683 25244 slave.cpp:964] Detecting new master [17:58:34]W: [Step 10/10] I0324 17:58:34.532719 25241 authenticatee.cpp:121] Creating new client SASL connection [17:58:34]W: [Step 10/10] I0324 17:58:34.532836 25244 slave.cpp:4912] Received oversubscribable resources from the resource estimator [17:58:34]W: [Step 10/10] I0324 17:58:34.532969 25245 master.cpp:5679] Authenticating slave(57)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.533061 25247 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(159)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.533257 25244 authenticator.cpp:98] Creating new server SASL connection [17:58:34]W: [Step 10/10] I0324 17:58:34.533473 25243 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 [17:58:34]W: [Step 10/10] I0324 17:58:34.533500 25243 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' [17:58:34]W: [Step 10/10] I0324 17:58:34.533601 25241 authenticator.cpp:203] Received SASL authentication start [17:58:34]W: [Step 10/10] I0324 17:58:34.533644 25241 authenticator.cpp:325] Authentication requires more steps [17:58:34]W: [Step 10/10] I0324 17:58:34.533730 25241 authenticatee.cpp:258] Received SASL authentication step [17:58:34]W: [Step 10/10] I0324 17:58:34.533820 25243 authenticator.cpp:231] Received SASL authentication step [17:58:34]W: [Step 10/10] I0324 17:58:34.533843 25243 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-190.mesosphere.io' server FQDN: 'ip-172-30-2-190.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [17:58:34]W: [Step 10/10] I0324 17:58:34.533856 25243 auxprop.cpp:179] Looking up auxiliary property '*userPassword' [17:58:34]W: [Step 10/10] I0324 17:58:34.533893 25243 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [17:58:34]W: [Step 10/10] I0324 17:58:34.533916 25243 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-190.mesosphere.io' server FQDN: 'ip-172-30-2-190.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [17:58:34]W: [Step 10/10] I0324 17:58:34.533924 25243 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [17:58:34]W: [Step 10/10] I0324 17:58:34.533929 25243 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [17:58:34]W: [Step 10/10] I0324 17:58:34.533939 25243 authenticator.cpp:317] Authentication success [17:58:34]W: [Step 10/10] I0324 17:58:34.534014 25241 authenticatee.cpp:298] Authentication success [17:58:34]W: [Step 10/10] I0324 17:58:34.534049 25244 master.cpp:5709] Successfully authenticated principal 'test-principal' at slave(57)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.534085 25248 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(159)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.534303 25246 slave.cpp:1061] Successfully authenticated with master master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.534396 25246 slave.cpp:1457] Will retry registration in 2.330159ms if necessary [17:58:34]W: [Step 10/10] I0324 17:58:34.534548 25241 master.cpp:4390] Registering slave at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) with id 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:34]W: [Step 10/10] I0324 17:58:34.534890 25245 registrar.cpp:439] Applied 1 operations in 46197ns; attempting to update the 'registry' [17:58:34]W: [Step 10/10] I0324 17:58:34.535507 25247 log.cpp:683] Attempting to append 396 bytes to the log [17:58:34]W: [Step 10/10] I0324 17:58:34.535614 25241 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 [17:58:34]W: [Step 10/10] I0324 17:58:34.536226 25247 replica.cpp:537] Replica received write request for position 3 from (2705)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.536916 25247 leveldb.cpp:341] Persisting action (415 bytes) to leveldb took 663688ns [17:58:34]W: [Step 10/10] I0324 17:58:34.536934 25247 replica.cpp:712] Persisted action at 3 [17:58:34]W: [Step 10/10] I0324 17:58:34.537461 25248 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 [17:58:34]W: [Step 10/10] I0324 17:58:34.537940 25243 slave.cpp:1457] Will retry registration in 9.623865ms if necessary [17:58:34]W: [Step 10/10] I0324 17:58:34.538035 25246 master.cpp:4378] Ignoring register slave message from slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) as admission is already in progress [17:58:34]W: [Step 10/10] I0324 17:58:34.539789 25248 leveldb.cpp:341] Persisting action (417 bytes) to leveldb took 2.306533ms [17:58:34]W: [Step 10/10] I0324 17:58:34.539813 25248 replica.cpp:712] Persisted action at 3 [17:58:34]W: [Step 10/10] I0324 17:58:34.539826 25248 replica.cpp:697] Replica learned APPEND action at position 3 [17:58:34]W: [Step 10/10] I0324 17:58:34.540894 25243 registrar.cpp:484] Successfully updated the 'registry' in 5.949184ms [17:58:34]W: [Step 10/10] I0324 17:58:34.541115 25248 log.cpp:702] Attempting to truncate the log to 3 [17:58:34]W: [Step 10/10] I0324 17:58:34.541218 25243 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 [17:58:34]W: [Step 10/10] I0324 17:58:34.541563 25246 slave.cpp:3618] Received ping from slave-observer(55)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.541703 25244 master.cpp:4458] Registered slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] [17:58:34]W: [Step 10/10] I0324 17:58:34.541940 25242 slave.cpp:1105] Registered with master master@172.30.2.190:33501; given slave ID 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:34]W: [Step 10/10] I0324 17:58:34.541956 25241 replica.cpp:537] Replica received write request for position 4 from (2706)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.541965 25242 fetcher.cpp:81] Clearing fetcher cache [17:58:34]W: [Step 10/10] I0324 17:58:34.541960 25243 hierarchical.cpp:473] Added slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 (ip-172-30-2-190.mesosphere.io) with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] (allocated: ) [17:58:34]W: [Step 10/10] I0324 17:58:34.542150 25244 status_update_manager.cpp:181] Resuming sending status updates [17:58:34]W: [Step 10/10] I0324 17:58:34.542153 25243 hierarchical.cpp:1487] No resources available to allocate! [17:58:34]W: [Step 10/10] I0324 17:58:34.542217 25243 hierarchical.cpp:1161] Performed allocation for slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 in 195303ns [17:58:34]W: [Step 10/10] I0324 17:58:34.542264 25242 slave.cpp:1128] Checkpointing SlaveInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/slave.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.542471 25242 slave.cpp:1165] Forwarding total oversubscribed resources [17:58:34]W: [Step 10/10] I0324 17:58:34.542592 25242 master.cpp:4802] Received update of slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) with total oversubscribed resources [17:58:34]W: [Step 10/10] I0324 17:58:34.542688 25227 sched.cpp:222] Version: 0.29.0 [17:58:34]W: [Step 10/10] I0324 17:58:34.542974 25242 hierarchical.cpp:531] Slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 (ip-172-30-2-190.mesosphere.io) updated with oversubscribed resources (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: ) [17:58:34]W: [Step 10/10] I0324 17:58:34.543114 25243 sched.cpp:326] New master detected at master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.543113 25242 hierarchical.cpp:1487] No resources available to allocate! [17:58:34]W: [Step 10/10] I0324 17:58:34.543220 25242 hierarchical.cpp:1161] Performed allocation for slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 in 208288ns [17:58:34]W: [Step 10/10] I0324 17:58:34.543231 25243 sched.cpp:382] Authenticating with master master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.543247 25243 sched.cpp:389] Using default CRAM-MD5 authenticatee [17:58:34]W: [Step 10/10] I0324 17:58:34.543412 25247 authenticatee.cpp:121] Creating new client SASL connection [17:58:34]W: [Step 10/10] I0324 17:58:34.543650 25244 master.cpp:5679] Authenticating scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.543731 25243 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(160)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.543898 25245 authenticator.cpp:98] Creating new server SASL connection [17:58:34]W: [Step 10/10] I0324 17:58:34.544088 25248 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 [17:58:34]W: [Step 10/10] I0324 17:58:34.544114 25248 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' [17:58:34]W: [Step 10/10] I0324 17:58:34.544188 25248 authenticator.cpp:203] Received SASL authentication start [17:58:34]W: [Step 10/10] I0324 17:58:34.544224 25248 authenticator.cpp:325] Authentication requires more steps [17:58:34]W: [Step 10/10] I0324 17:58:34.544296 25248 authenticatee.cpp:258] Received SASL authentication step [17:58:34]W: [Step 10/10] I0324 17:58:34.544322 25241 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 2.331864ms [17:58:34]W: [Step 10/10] I0324 17:58:34.544349 25241 replica.cpp:712] Persisted action at 4 [17:58:34]W: [Step 10/10] I0324 17:58:34.544385 25248 authenticator.cpp:231] Received SASL authentication step [17:58:34]W: [Step 10/10] I0324 17:58:34.544412 25248 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-190.mesosphere.io' server FQDN: 'ip-172-30-2-190.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [17:58:34]W: [Step 10/10] I0324 17:58:34.544422 25248 auxprop.cpp:179] Looking up auxiliary property '*userPassword' [17:58:34]W: [Step 10/10] I0324 17:58:34.544456 25248 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [17:58:34]W: [Step 10/10] I0324 17:58:34.544479 25248 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-190.mesosphere.io' server FQDN: 'ip-172-30-2-190.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [17:58:34]W: [Step 10/10] I0324 17:58:34.544487 25248 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [17:58:34]W: [Step 10/10] I0324 17:58:34.544492 25248 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [17:58:34]W: [Step 10/10] I0324 17:58:34.544502 25248 authenticator.cpp:317] Authentication success [17:58:34]W: [Step 10/10] I0324 17:58:34.544672 25242 authenticatee.cpp:298] Authentication success [17:58:34]W: [Step 10/10] I0324 17:58:34.544734 25246 master.cpp:5709] Successfully authenticated principal 'test-principal' at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.544755 25241 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(160)@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.544917 25243 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 [17:58:34]W: [Step 10/10] I0324 17:58:34.545156 25248 sched.cpp:472] Successfully authenticated with master master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.545173 25248 sched.cpp:777] Sending SUBSCRIBE call to master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.545241 25248 sched.cpp:810] Will retry registration in 154.057742ms if necessary [17:58:34]W: [Step 10/10] I0324 17:58:34.545344 25247 master.cpp:2346] Received SUBSCRIBE call for framework 'default' at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:34]W: [Step 10/10] W0324 17:58:34.545368 25247 master.cpp:2353] Framework at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 (authenticated as 'test-principal') does not set 'principal' in FrameworkInfo [17:58:34]W: [Step 10/10] I0324 17:58:34.545403 25247 master.cpp:1865] Authorizing framework principal '' to receive offers for role '*' [17:58:34]W: [Step 10/10] I0324 17:58:34.545600 25242 master.cpp:2417] Subscribing framework default with checkpointing enabled and capabilities [ ] [17:58:34]W: [Step 10/10] I0324 17:58:34.545903 25246 hierarchical.cpp:265] Added framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.545986 25248 sched.cpp:704] Framework registered with 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.546020 25248 sched.cpp:718] Scheduler::registered took 16098ns [17:58:34]W: [Step 10/10] I0324 17:58:34.546480 25246 hierarchical.cpp:1582] No inverse offers to send out! [17:58:34]W: [Step 10/10] I0324 17:58:34.546541 25246 hierarchical.cpp:1138] Performed allocation for 1 slaves in 617704ns [17:58:34]W: [Step 10/10] I0324 17:58:34.546736 25242 master.cpp:5508] Sending 1 offers to framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.546815 25243 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 1.871252ms [17:58:34]W: [Step 10/10] I0324 17:58:34.546875 25243 leveldb.cpp:399] Deleting ~2 keys from leveldb took 35863ns [17:58:34]W: [Step 10/10] I0324 17:58:34.546897 25243 replica.cpp:712] Persisted action at 4 [17:58:34]W: [Step 10/10] I0324 17:58:34.546921 25243 replica.cpp:697] Replica learned TRUNCATE action at position 4 [17:58:34]W: [Step 10/10] I0324 17:58:34.547122 25245 sched.cpp:874] Scheduler::resourceOffers took 122109ns [17:58:34]W: [Step 10/10] I0324 17:58:34.547441 25227 resources.cpp:572] Parsing resources as JSON failed: cpus:1;mem:1 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.547608 25227 resources.cpp:572] Parsing resources as JSON failed: cpus:1;mem:1 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.547765 25227 resources.cpp:572] Parsing resources as JSON failed: cpus:1;mem:1 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.548921 25242 master.cpp:3288] Processing ACCEPT call for offers: [ 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-O0 ] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.548956 25242 master.cpp:2891] Authorizing framework principal '' to launch task 0 as user 'root' [17:58:34]W: [Step 10/10] I0324 17:58:34.549037 25242 master.cpp:2891] Authorizing framework principal '' to launch task 1 as user 'root' [17:58:34]W: [Step 10/10] I0324 17:58:34.549100 25242 master.cpp:2891] Authorizing framework principal '' to launch task 2 as user 'root' [17:58:34]W: [Step 10/10] I0324 17:58:34.550242 25244 master.hpp:177] Adding task 0 with resources cpus(*):1; mem(*):1 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.550325 25244 master.cpp:3773] Launching task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 with resources cpus(*):1; mem(*):1 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.550570 25245 slave.cpp:1497] Got assigned task 0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.550662 25245 slave.cpp:5448] Checkpointing FrameworkInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/framework.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.550837 25245 slave.cpp:5459] Checkpointing framework pid 'scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/framework.pid' [17:58:34]W: [Step 10/10] I0324 17:58:34.550843 25244 master.hpp:177] Adding task 1 with resources cpus(*):1; mem(*):1 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.550918 25244 master.cpp:3773] Launching task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 with resources cpus(*):1; mem(*):1 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.551004 25245 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.551435 25245 slave.cpp:1497] Got assigned task 1 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.551424 25244 master.hpp:177] Adding task 2 with resources cpus(*):1; mem(*):1 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.551498 25245 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.551499 25244 master.cpp:3773] Launching task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 with resources cpus(*):1; mem(*):1 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.551847 25245 slave.cpp:1616] Launching task 0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.551904 25245 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.552005 25248 hierarchical.cpp:890] Recovered cpus(*):997; mem(*):997; disk(*):3962; ports(*):[31000-32000] (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: cpus(*):3; mem(*):3) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.552048 25248 hierarchical.cpp:927] Framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 filtered slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for 5secs [17:58:34]W: [Step 10/10] I0324 17:58:34.552265 25245 paths.cpp:528] Trying to chown '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89' to user 'root' [17:58:34]W: [Step 10/10] I0324 17:58:34.554826 25245 slave.cpp:5900] Checkpointing ExecutorInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/executor.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.555132 25245 slave.cpp:5528] Launching executor 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89' [17:58:34]W: [Step 10/10] I0324 17:58:34.555537 25244 containerizer.cpp:666] Starting container '892d0f4a-6e99-470a-a85b-52e2bbcb7e89' for executor '0' of framework '9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000' [17:58:34]W: [Step 10/10] I0324 17:58:34.555591 25245 slave.cpp:5923] Checkpointing TaskInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/tasks/0/task.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.555794 25245 slave.cpp:1834] Queuing task '0' for executor '0' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.555948 25245 slave.cpp:1497] Got assigned task 2 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.556010 25245 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.556380 25245 slave.cpp:1616] Launching task 1 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.556437 25245 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.556715 25245 paths.cpp:528] Trying to chown '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' to user 'root' [17:58:34]W: [Step 10/10] I0324 17:58:34.558962 25245 slave.cpp:5900] Checkpointing ExecutorInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/executor.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.559285 25245 slave.cpp:5528] Launching executor 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' [17:58:34]W: [Step 10/10] I0324 17:58:34.559692 25245 slave.cpp:5923] Checkpointing TaskInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/tasks/1/task.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.559882 25245 slave.cpp:1834] Queuing task '1' for executor '1' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.559963 25245 slave.cpp:881] Successfully attached file '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89' [17:58:34]W: [Step 10/10] I0324 17:58:34.560000 25245 slave.cpp:1616] Launching task 2 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.560055 25245 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [17:58:34]W: [Step 10/10] Trying semicolon-delimited string format instead [17:58:34]W: [Step 10/10] I0324 17:58:34.560359 25245 paths.cpp:528] Trying to chown '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' to user 'root' [17:58:34]W: [Step 10/10] I0324 17:58:34.560389 25244 linux_launcher.cpp:304] Cloning child process with flags = [17:58:34]W: [Step 10/10] I0324 17:58:34.562607 25244 containerizer.cpp:1147] Checkpointing executor's forked pid 29133 to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/pids/forked.pid' [17:58:34]W: [Step 10/10] I0324 17:58:34.562729 25245 slave.cpp:5900] Checkpointing ExecutorInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/executor.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.563026 25245 slave.cpp:5528] Launching executor 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' [17:58:34]W: [Step 10/10] I0324 17:58:34.563400 25245 slave.cpp:5923] Checkpointing TaskInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7/tasks/2/task.info' [17:58:34]W: [Step 10/10] I0324 17:58:34.563585 25245 slave.cpp:1834] Queuing task '2' for executor '2' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.563640 25244 containerizer.cpp:666] Starting container 'bd979c82-be7b-48b4-931b-26c848e8b592' for executor '1' of framework '9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000' [17:58:34]W: [Step 10/10] I0324 17:58:34.563660 25245 slave.cpp:881] Successfully attached file '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' [17:58:34]W: [Step 10/10] I0324 17:58:34.563691 25245 slave.cpp:881] Successfully attached file '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' [17:58:34]W: [Step 10/10] I0324 17:58:34.565599 25244 containerizer.cpp:666] Starting container '86be5a57-b217-4535-8b84-d3709100f1a7' for executor '2' of framework '9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000' [17:58:34]W: [Step 10/10] I0324 17:58:34.568126 25243 fetcher.cpp:323] Starting to fetch URIs for container: 892d0f4a-6e99-470a-a85b-52e2bbcb7e89, directory: /tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89 [17:58:34]W: [Step 10/10] I0324 17:58:34.570322 25243 fetcher.cpp:912] Created cache entry 'root@http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' with file: c1-mesos-fetc_rchive.tgz [17:58:34]W: [Step 10/10] I0324 17:58:34.571252 25248 fetcher.cpp:271] Fetching size for URI: http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz [17:58:34]W: [Step 10/10] I0324 17:58:34.571570 25244 linux_launcher.cpp:304] Cloning child process with flags = [17:58:34]W: [Step 10/10] I0324 17:58:34.574787 25244 containerizer.cpp:1147] Checkpointing executor's forked pid 29150 to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/pids/forked.pid' [17:58:34]W: [Step 10/10] I0324 17:58:34.577056 25244 linux_launcher.cpp:304] Cloning child process with flags = [17:58:34]W: [Step 10/10] I0324 17:58:34.578572 25244 containerizer.cpp:1147] Checkpointing executor's forked pid 29154 to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7/pids/forked.pid' [17:58:34]W: [Step 10/10] I0324 17:58:34.580124 25243 fetcher.cpp:323] Starting to fetch URIs for container: bd979c82-be7b-48b4-931b-26c848e8b592, directory: /tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592 [17:58:34]W: [Step 10/10] I0324 17:58:34.585125 25243 fetcher.cpp:323] Starting to fetch URIs for container: 86be5a57-b217-4535-8b84-d3709100f1a7, directory: /tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7 [17:58:34]W: [Step 10/10] I0324 17:58:34.587946 25247 process.cpp:3136] Handling HTTP event for process '(2689)' with path: '/(2689)/mesos-fetcher-test-archive.tgz' [17:58:34]W: [Step 10/10] I0324 17:58:34.588325 25247 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/mesos-fetcher-test-assets/mesos-fetcher-test-archive.tgz' with length 10240 [17:58:34] : [Step 10/10] HTTP/1.1 200 OK [17:58:34] : [Step 10/10] Date: Thu, 24 Mar 2016 17:58:34 GMT [17:58:34] : [Step 10/10] Content-Length: 10240 [17:58:34] : [Step 10/10] [17:58:34]W: [Step 10/10] I0324 17:58:34.591938 25243 fetcher.cpp:785] Fetching URIs using command '/mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-fetcher' [17:58:34]W: [Step 10/10] I0324 17:58:34.593366 25243 fetcher.cpp:687] Claiming fetcher cache space for: root@http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz [17:58:34]W: [Step 10/10] I0324 17:58:34.593385 25243 fetcher.cpp:1132] Claimed cache space: 10KB, now using: 10KB [17:58:34]W: [Step 10/10] I0324 17:58:34.597213 25243 fetcher.cpp:785] Fetching URIs using command '/mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-fetcher' [17:58:34]W: [Step 10/10] I0324 17:58:34.650645 25245 process.cpp:3136] Handling HTTP event for process '(2689)' with path: '/(2689)/mesos-fetcher-test-archive.tgz' [17:58:34]W: [Step 10/10] I0324 17:58:34.650964 25243 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/mesos-fetcher-test-assets/mesos-fetcher-test-archive.tgz' with length 10240 [17:58:34]W: [Step 10/10] I0324 17:58:34.651347 25246 process.cpp:3136] Handling HTTP event for process '(2689)' with path: '/(2689)/mesos-fetcher-test-archive.tgz' [17:58:34]W: [Step 10/10] I0324 17:58:34.651589 25242 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/mesos-fetcher-test-assets/mesos-fetcher-test-archive.tgz' with length 10240 [17:58:34]W: [Step 10/10] I0324 17:58:34.654781 25241 process.cpp:3136] Handling HTTP event for process '(2689)' with path: '/(2689)/mesos-fetcher-test-cmd' [17:58:34]W: [Step 10/10] I0324 17:58:34.654969 25242 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/mesos-fetcher-test-assets/mesos-fetcher-test-cmd' with length 30 [17:58:34]W: [Step 10/10] I0324 17:58:34.655164 25244 process.cpp:3136] Handling HTTP event for process '(2689)' with path: '/(2689)/mesos-fetcher-test-cmd' [17:58:34]W: [Step 10/10] I0324 17:58:34.655402 25248 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/mesos-fetcher-test-assets/mesos-fetcher-test-cmd' with length 30 [17:58:34]W: [Step 10/10] I0324 17:58:34.721863 25241 fetcher.cpp:1144] Released cache space: 0B, now using: 10KB [17:58:34]W: [Step 10/10] I0324 17:58:34.726083 25243 fetcher.cpp:785] Fetching URIs using command '/mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-fetcher' [17:58:34]W: [Step 10/10] WARNING: Logging before InitGoogleLogging() is written to STDERR [17:58:34]W: [Step 10/10] I0324 17:58:34.774386 29226 process.cpp:986] libprocess is initialized on 172.30.2.190:54429 for 8 cpus [17:58:34]W: [Step 10/10] WARNING: Logging before InitGoogleLogging() is written to STDERR [17:58:34]W: [Step 10/10] I0324 17:58:34.774593 29225 process.cpp:986] libprocess is initialized on 172.30.2.190:47691 for 8 cpus [17:58:34]W: [Step 10/10] I0324 17:58:34.775140 29225 logging.cpp:195] Logging to STDERR [17:58:34]W: [Step 10/10] I0324 17:58:34.777151 29225 exec.cpp:150] Version: 0.29.0 [17:58:34]W: [Step 10/10] I0324 17:58:34.779659 29226 logging.cpp:195] Logging to STDERR [17:58:34]W: [Step 10/10] I0324 17:58:34.781606 29226 exec.cpp:150] Version: 0.29.0 [17:58:34]W: [Step 10/10] I0324 17:58:34.781774 25242 process.cpp:3136] Handling HTTP event for process '(2689)' with path: '/(2689)/mesos-fetcher-test-cmd' [17:58:34]W: [Step 10/10] I0324 17:58:34.782081 25244 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/mesos-fetcher-test-assets/mesos-fetcher-test-cmd' with length 30 [17:58:34]W: [Step 10/10] I0324 17:58:34.783602 29278 exec.cpp:200] Executor started at: executor(1)@172.30.2.190:47691 with pid 29225 [17:58:34]W: [Step 10/10] I0324 17:58:34.784312 29280 exec.cpp:200] Executor started at: executor(1)@172.30.2.190:54429 with pid 29226 [17:58:34]W: [Step 10/10] I0324 17:58:34.785394 25246 slave.cpp:2779] Got registration for executor '0' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:47691 [17:58:34]W: [Step 10/10] I0324 17:58:34.785792 25246 slave.cpp:2865] Checkpointing executor pid 'executor(1)@172.30.2.190:47691' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/pids/libprocess.pid' [17:58:34]W: [Step 10/10] I0324 17:58:34.786747 25246 slave.cpp:2779] Got registration for executor '1' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:54429 [17:58:34]W: [Step 10/10] I0324 17:58:34.787050 25246 slave.cpp:2865] Checkpointing executor pid 'executor(1)@172.30.2.190:54429' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/pids/libprocess.pid' [17:58:34]W: [Step 10/10] I0324 17:58:34.787273 29281 exec.cpp:225] Executor registered on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:34]W: [Step 10/10] I0324 17:58:34.787750 25246 slave.cpp:1999] Sending queued task '0' to executor '0' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 at executor(1)@172.30.2.190:47691 [17:58:34]W: [Step 10/10] I0324 17:58:34.788139 29284 exec.cpp:225] Executor registered on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:34]W: [Step 10/10] I0324 17:58:34.788450 25248 slave.cpp:1999] Sending queued task '1' to executor '1' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 at executor(1)@172.30.2.190:54429 [17:58:34]W: [Step 10/10] I0324 17:58:34.788959 29281 exec.cpp:237] Executor::registered took 293219ns [17:58:34]W: [Step 10/10] I0324 17:58:34.789197 29281 exec.cpp:312] Executor asked to run task '0' [17:58:34]W: [Step 10/10] I0324 17:58:34.789286 29281 exec.cpp:321] Executor::launchTask took 70432ns [17:58:34] : [Step 10/10] Registered executor on ip-172-30-2-190.mesosphere.io [17:58:34] : [Step 10/10] Starting task 0 [17:58:34]W: [Step 10/10] I0324 17:58:34.789826 29284 exec.cpp:237] Executor::registered took 272773ns [17:58:34]W: [Step 10/10] I0324 17:58:34.790019 29284 exec.cpp:312] Executor asked to run task '1' [17:58:34]W: [Step 10/10] I0324 17:58:34.790109 29284 exec.cpp:321] Executor::launchTask took 70411ns [17:58:34]W: [Step 10/10] I0324 17:58:34.791676 29284 exec.cpp:535] Executor sending status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.792073 29279 exec.cpp:535] Executor sending status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.793035 25248 slave.cpp:3138] Handling status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:54429 [17:58:34]W: [Step 10/10] I0324 17:58:34.793324 25248 slave.cpp:3138] Handling status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:47691 [17:58:34]W: [Step 10/10] I0324 17:58:34.794428 25241 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.794476 25241 status_update_manager.cpp:497] Creating StatusUpdate stream for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34] : [Step 10/10] Forked command at 29291 [17:58:34] : [Step 10/10] sh -c './mesos-fetcher-test-cmd 0' [17:58:34] : [Step 10/10] Registered executor on ip-172-30-2-190.mesosphere.io [17:58:34] : [Step 10/10] Starting task 1 [17:58:34] : [Step 10/10] sh -c './mesos-fetcher-test-acmd 1' [17:58:34] : [Step 10/10] Forked command at 29293 [17:58:34]W: [Step 10/10] I0324 17:58:34.794843 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.799917 25241 status_update_manager.cpp:374] Forwarding update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to the slave [17:58:34]W: [Step 10/10] I0324 17:58:34.800144 25246 slave.cpp:3536] Forwarding the update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to master@172.30.2.190:33501 [17:58:34]W: [Step 10/10] I0324 17:58:34.800212 25241 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.800248 25241 status_update_manager.cpp:497] Creating StatusUpdate stream for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.800304 25246 slave.cpp:3430] Status update manager successfully handled status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.800351 25246 slave.cpp:3446] Sending acknowledgement for status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to executor(1)@172.30.2.190:54429 [17:58:34]W: [Step 10/10] I0324 17:58:34.800459 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.800484 25248 master.cpp:4947] Status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:34]W: [Step 10/10] I0324 17:58:34.800531 25248 master.cpp:4995] Forwarding status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.800709 25248 master.cpp:6608] Updating the state of task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) [17:58:34]W: [Step 10/10] I0324 17:58:34.800941 29270 exec.cpp:358] Executor received status update acknowledgement bf91655d-7f65-42a4-8497-b6ecea80605c for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:34]W: [Step 10/10] I0324 17:58:34.801031 25245 sched.cpp:982] Scheduler::statusUpdate took 209043ns [17:58:34]W: [Step 10/10] I0324 17:58:34.801240 25245 master.cpp:4102] Processing ACKNOWLEDGE call bf91655d-7f65-42a4-8497-b6ecea80605c for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:34]W: [Step 10/10] WARNING: Logging before InitGoogleLogging() is written to STDERR [17:58:34] : [Step 10/10] Command exited with status 0 (pid: 29291) [17:58:36]W: [Step 10/10] I0324 17:58:34.870443 29297 process.cpp:986] libprocess is initialized on 172.30.2.190:42013 for 8 cpus [17:58:36] : [Step 10/10] Command exited with status 0 (pid: 29293) [17:58:36]W: [Step 10/10] I0324 17:58:34.871054 29297 logging.cpp:195] Logging to STDERR [17:58:36]W: [Step 10/10] I0324 17:58:34.872535 29297 exec.cpp:150] Version: 0.29.0 [17:58:36]W: [Step 10/10] I0324 17:58:34.877939 29318 exec.cpp:200] Executor started at: executor(1)@172.30.2.190:42013 with pid 29297 [17:58:36]W: [Step 10/10] I0324 17:58:34.879621 25248 slave.cpp:2779] Got registration for executor '2' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:42013 [17:58:36]W: [Step 10/10] I0324 17:58:34.880053 25248 slave.cpp:2865] Checkpointing executor pid 'executor(1)@172.30.2.190:42013' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7/pids/libprocess.pid' [17:58:36]W: [Step 10/10] I0324 17:58:34.892945 29269 exec.cpp:535] Executor sending status update TASK_FINISHED (UUID: 1d517754-d0d0-4f00-9797-3bf8a25e8e6e) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:34.894079 29273 exec.cpp:535] Executor sending status update TASK_FINISHED (UUID: 6e41ae85-62a9-4dba-97ec-46a393e8f78f) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:35.415397 25242 hierarchical.cpp:1673] Filtered offer with cpus(*):997; mem(*):997; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:35.415443 25242 hierarchical.cpp:1487] No resources available to allocate! [17:58:36]W: [Step 10/10] I0324 17:58:35.415480 25242 hierarchical.cpp:1582] No inverse offers to send out! [17:58:36]W: [Step 10/10] I0324 17:58:35.415546 25242 hierarchical.cpp:1138] Performed allocation for 1 slaves in 695812ns [17:58:36]W: [Step 10/10] I0324 17:58:35.929672 25246 containerizer.cpp:1656] Executor for container '892d0f4a-6e99-470a-a85b-52e2bbcb7e89' has exited [17:58:36]W: [Step 10/10] I0324 17:58:35.929700 25246 containerizer.cpp:1421] Destroying container '892d0f4a-6e99-470a-a85b-52e2bbcb7e89' [17:58:36]W: [Step 10/10] I0324 17:58:35.931381 25242 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/892d0f4a-6e99-470a-a85b-52e2bbcb7e89 [17:58:36]W: [Step 10/10] I0324 17:58:35.931427 25246 containerizer.cpp:1656] Executor for container 'bd979c82-be7b-48b4-931b-26c848e8b592' has exited [17:58:36]W: [Step 10/10] I0324 17:58:35.931457 25246 containerizer.cpp:1421] Destroying container 'bd979c82-be7b-48b4-931b-26c848e8b592' [17:58:36]W: [Step 10/10] I0324 17:58:35.933028 25243 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/892d0f4a-6e99-470a-a85b-52e2bbcb7e89 after 1.593856ms [17:58:36]W: [Step 10/10] I0324 17:58:35.933073 25246 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/bd979c82-be7b-48b4-931b-26c848e8b592 [17:58:36]W: [Step 10/10] I0324 17:58:35.934610 25242 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/bd979c82-be7b-48b4-931b-26c848e8b592 after 1.487872ms [17:58:36]W: [Step 10/10] I0324 17:58:35.934842 25245 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/892d0f4a-6e99-470a-a85b-52e2bbcb7e89 [17:58:36]W: [Step 10/10] I0324 17:58:35.936211 25246 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/bd979c82-be7b-48b4-931b-26c848e8b592 [17:58:36]W: [Step 10/10] I0324 17:58:35.936391 25247 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/892d0f4a-6e99-470a-a85b-52e2bbcb7e89 after 1.496064ms [17:58:36]W: [Step 10/10] I0324 17:58:35.937597 25246 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/bd979c82-be7b-48b4-931b-26c848e8b592 after 1.345024ms [17:58:36]W: [Step 10/10] I0324 17:58:35.939069 25247 provisioner.cpp:306] Ignoring destroy request for unknown container 892d0f4a-6e99-470a-a85b-52e2bbcb7e89 [17:58:36]W: [Step 10/10] I0324 17:58:35.940606 25244 provisioner.cpp:306] Ignoring destroy request for unknown container bd979c82-be7b-48b4-931b-26c848e8b592 [17:58:36]W: [Step 10/10] I0324 17:58:36.417083 25244 hierarchical.cpp:1673] Filtered offer with cpus(*):997; mem(*):997; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.417119 25244 hierarchical.cpp:1487] No resources available to allocate! [17:58:36]W: [Step 10/10] I0324 17:58:36.417147 25244 hierarchical.cpp:1582] No inverse offers to send out! [17:58:36]W: [Step 10/10] I0324 17:58:36.417196 25244 hierarchical.cpp:1138] Performed allocation for 1 slaves in 489951ns [17:58:36] : [Step 10/10] Registered executor on ip-172-30-2-190.mesosphere.io [17:58:36]W: [Step 10/10] I0324 17:58:36.798274 25248 slave.cpp:3138] Handling status update TASK_FINISHED (UUID: 1d517754-d0d0-4f00-9797-3bf8a25e8e6e) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:47691 [17:58:36]W: [Step 10/10] I0324 17:58:36.798673 25248 slave.cpp:3138] Handling status update TASK_FINISHED (UUID: 6e41ae85-62a9-4dba-97ec-46a393e8f78f) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:54429 [17:58:36]W: [Step 10/10] I0324 17:58:36.798745 25241 status_update_manager.cpp:374] Forwarding update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to the slave [17:58:36]W: [Step 10/10] I0324 17:58:36.798908 29314 exec.cpp:225] Executor registered on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:36]W: [Step 10/10] I0324 17:58:36.798945 25248 slave.cpp:3664] executor(1)@172.30.2.190:54429 exited [17:58:36]W: [Step 10/10] I0324 17:58:36.798986 25248 slave.cpp:3664] executor(1)@172.30.2.190:47691 exited [17:58:36]W: [Step 10/10] I0324 17:58:36.799028 25248 slave.cpp:4039] Executor '0' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 exited with status 0 [17:58:36]W: [Step 10/10] I0324 17:58:36.799094 25241 status_update_manager.cpp:392] Received status update acknowledgement (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.799127 25248 slave.cpp:3138] Handling status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from @0.0.0.0:0 [17:58:36]W: [Step 10/10] I0324 17:58:36.799235 25241 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36] : [Step 10/10] Starting task 2 [17:58:36] : [Step 10/10] sh -c './mesos-fetcher-test-acmd 2' [17:58:36]W: [Step 10/10] I0324 17:58:36.799381 25248 slave.cpp:4039] Executor '1' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 exited with status 0 [17:58:36] : [Step 10/10] Forked command at 29320 [17:58:36]W: [Step 10/10] I0324 17:58:36.799468 25248 slave.cpp:3138] Handling status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from @0.0.0.0:0 [17:58:36]W: [Step 10/10] I0324 17:58:36.799823 25248 slave.cpp:1999] Sending queued task '2' to executor '2' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 at executor(1)@172.30.2.190:42013 [17:58:36]W: [Step 10/10] I0324 17:58:36.800151 25248 slave.cpp:5838] Terminating task 0 [17:58:36]W: [Step 10/10] W0324 17:58:36.800406 25243 containerizer.cpp:1263] Ignoring update for unknown container: 892d0f4a-6e99-470a-a85b-52e2bbcb7e89 [17:58:36]W: [Step 10/10] I0324 17:58:36.800516 25248 slave.cpp:3536] Forwarding the update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to master@172.30.2.190:33501 [17:58:36]W: [Step 10/10] I0324 17:58:36.800773 25248 slave.cpp:5838] Terminating task 1 [17:58:36]W: [Step 10/10] I0324 17:58:36.800843 29314 exec.cpp:237] Executor::registered took 392129ns [17:58:36]W: [Step 10/10] I0324 17:58:36.800911 25245 master.cpp:4947] Status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.800957 25245 master.cpp:4995] Forwarding status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] W0324 17:58:36.801050 25242 containerizer.cpp:1263] Ignoring update for unknown container: bd979c82-be7b-48b4-931b-26c848e8b592 [17:58:36]W: [Step 10/10] I0324 17:58:36.801101 25248 slave.cpp:3430] Status update manager successfully handled status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.801157 25245 master.cpp:6608] Updating the state of task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (latest state: TASK_FINISHED, status update state: TASK_RUNNING) [17:58:36]W: [Step 10/10] I0324 17:58:36.801149 25248 slave.cpp:3446] Sending acknowledgement for status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to executor(1)@172.30.2.190:47691 [17:58:36]W: [Step 10/10] I0324 17:58:36.801192 29314 exec.cpp:312] Executor asked to run task '2' [17:58:36]W: [Step 10/10] I0324 17:58:36.801333 29314 exec.cpp:321] Executor::launchTask took 115598ns [17:58:36]W: [Step 10/10] I0324 17:58:36.801369 25243 sched.cpp:982] Scheduler::statusUpdate took 129150ns [17:58:36]W: [Step 10/10] I0324 17:58:36.801508 25249 poll_socket.cpp:110] Socket error while connecting [17:58:36]W: [Step 10/10] I0324 17:58:36.801556 25249 process.cpp:1650] Failed to send 'mesos.internal.StatusUpdateAcknowledgementMessage' to '172.30.2.190:47691', connect: Socket error while connecting [17:58:36]W: [Step 10/10] I0324 17:58:36.801548 25247 hierarchical.cpp:890] Recovered cpus(*):1; mem(*):1 (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):2) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] E0324 17:58:36.801630 25249 process.cpp:1958] Failed to shutdown socket with fd 7: Transport endpoint is not connected [17:58:36]W: [Step 10/10] I0324 17:58:36.801666 25243 master.cpp:4102] Processing ACKNOWLEDGE call 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:36]W: [Step 10/10] I0324 17:58:36.803611 25247 slave.cpp:2548] Status update manager successfully handled status update acknowledgement (UUID: bf91655d-7f65-42a4-8497-b6ecea80605c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.803638 25241 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.803681 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.804510 29312 exec.cpp:535] Executor sending status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.805280 25243 slave.cpp:3138] Handling status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:42013 [17:58:36]W: [Step 10/10] I0324 17:58:36.808056 25241 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.808074 25248 slave.cpp:3430] Status update manager successfully handled status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.808097 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.812053 25241 status_update_manager.cpp:374] Forwarding update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to the slave [17:58:36]W: [Step 10/10] I0324 17:58:36.812237 25242 slave.cpp:3536] Forwarding the update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to master@172.30.2.190:33501 [17:58:36]W: [Step 10/10] I0324 17:58:36.812275 25241 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: 1d517754-d0d0-4f00-9797-3bf8a25e8e6e) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.812314 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: 1d517754-d0d0-4f00-9797-3bf8a25e8e6e) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.812371 25242 slave.cpp:3430] Status update manager successfully handled status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.812481 25243 master.cpp:4947] Status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.812513 25243 master.cpp:4995] Forwarding status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.812624 25243 master.cpp:6608] Updating the state of task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (latest state: TASK_FINISHED, status update state: TASK_FAILED) [17:58:36]W: [Step 10/10] I0324 17:58:36.812759 25248 sched.cpp:982] Scheduler::statusUpdate took 72712ns [17:58:36]W: [Step 10/10] I0324 17:58:36.812872 25245 hierarchical.cpp:890] Recovered cpus(*):1; mem(*):1 (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: cpus(*):1; mem(*):1) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.813050 25243 master.cpp:4102] Processing ACKNOWLEDGE call c8e77a0d-22b5-4614-a5c9-d1761baff29c for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:36]W: [Step 10/10] I0324 17:58:36.813091 25243 master.cpp:6674] Removing task 1 with resources cpus(*):1; mem(*):1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.816144 25246 slave.cpp:3430] Status update manager successfully handled status update TASK_FINISHED (UUID: 1d517754-d0d0-4f00-9797-3bf8a25e8e6e) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.816166 25241 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: 6e41ae85-62a9-4dba-97ec-46a393e8f78f) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.816184 25246 slave.cpp:3446] Sending acknowledgement for status update TASK_FINISHED (UUID: 1d517754-d0d0-4f00-9797-3bf8a25e8e6e) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to executor(1)@172.30.2.190:47691 [17:58:36]W: [Step 10/10] I0324 17:58:36.816212 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: 6e41ae85-62a9-4dba-97ec-46a393e8f78f) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.816428 25249 poll_socket.cpp:110] Socket error while connecting [17:58:36]W: [Step 10/10] I0324 17:58:36.816459 25249 process.cpp:1650] Failed to send 'mesos.internal.StatusUpdateAcknowledgementMessage' to '172.30.2.190:47691', connect: Socket error while connecting [17:58:36]W: [Step 10/10] E0324 17:58:36.816509 25249 process.cpp:1958] Failed to shutdown socket with fd 7: Transport endpoint is not connected [17:58:36]W: [Step 10/10] I0324 17:58:36.820000 25243 slave.cpp:3430] Status update manager successfully handled status update TASK_FINISHED (UUID: 6e41ae85-62a9-4dba-97ec-46a393e8f78f) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.820019 25241 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.820053 25243 slave.cpp:3446] Sending acknowledgement for status update TASK_FINISHED (UUID: 6e41ae85-62a9-4dba-97ec-46a393e8f78f) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to executor(1)@172.30.2.190:54429 [17:58:36]W: [Step 10/10] I0324 17:58:36.820133 25241 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.820292 25249 poll_socket.cpp:110] Socket error while connecting [17:58:36]W: [Step 10/10] I0324 17:58:36.820322 25249 process.cpp:1650] Failed to send 'mesos.internal.StatusUpdateAcknowledgementMessage' to '172.30.2.190:54429', connect: Socket error while connecting [17:58:36]W: [Step 10/10] E0324 17:58:36.820371 25249 process.cpp:1958] Failed to shutdown socket with fd 7: Transport endpoint is not connected [17:58:36]W: [Step 10/10] I0324 17:58:36.823626 25241 status_update_manager.cpp:374] Forwarding update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to the slave [17:58:36]W: [Step 10/10] I0324 17:58:36.823802 25248 slave.cpp:3536] Forwarding the update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to master@172.30.2.190:33501 [17:58:36]W: [Step 10/10] I0324 17:58:36.823840 25241 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.823873 25241 status_update_manager.cpp:497] Creating StatusUpdate stream for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.823928 25248 slave.cpp:2548] Status update manager successfully handled status update acknowledgement (UUID: 2eeeb2b4-0fd1-4d42-abfa-c93397415d0d) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.824040 25242 master.cpp:4947] Status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.824070 25242 master.cpp:4995] Forwarding status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.824183 25242 master.cpp:6608] Updating the state of task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (latest state: TASK_FINISHED, status update state: TASK_FAILED) [17:58:36]W: [Step 10/10] I0324 17:58:36.824190 25241 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.824337 25244 sched.cpp:982] Scheduler::statusUpdate took 66742ns [17:58:36]W: [Step 10/10] I0324 17:58:36.824530 25244 master.cpp:4102] Processing ACKNOWLEDGE call 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93 for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:36]W: [Step 10/10] I0324 17:58:36.824573 25244 master.cpp:6674] Removing task 0 with resources cpus(*):1; mem(*):1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.828418 25241 status_update_manager.cpp:374] Forwarding update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to the slave [17:58:36]W: [Step 10/10] I0324 17:58:36.828615 25244 slave.cpp:3536] Forwarding the update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to master@172.30.2.190:33501 [17:58:36]W: [Step 10/10] I0324 17:58:36.828696 25241 status_update_manager.cpp:392] Received status update acknowledgement (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.828765 25244 slave.cpp:3430] Status update manager successfully handled status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.828778 25241 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.828797 25244 slave.cpp:3446] Sending acknowledgement for status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to executor(1)@172.30.2.190:42013 [17:58:36]W: [Step 10/10] I0324 17:58:36.828891 25247 master.cpp:4947] Status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.828922 25247 master.cpp:4995] Forwarding status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.829061 25247 master.cpp:6608] Updating the state of task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) [17:58:36]W: [Step 10/10] I0324 17:58:36.829213 25246 sched.cpp:982] Scheduler::statusUpdate took 98886ns [17:58:36]W: [Step 10/10] I0324 17:58:36.829291 29317 exec.cpp:358] Executor received status update acknowledgement c69b2bb4-9439-4308-8ad8-f9099a36042f for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.829414 25242 master.cpp:4102] Processing ACKNOWLEDGE call c69b2bb4-9439-4308-8ad8-f9099a36042f for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:36]W: [Step 10/10] W0324 17:58:36.832485 25241 status_update_manager.cpp:446] Acknowledged a terminal status update TASK_FAILED (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 but updates are still pending [17:58:36]W: [Step 10/10] I0324 17:58:36.832512 25241 status_update_manager.cpp:528] Cleaning up status update stream for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.832736 25242 slave.cpp:2548] Status update manager successfully handled status update acknowledgement (UUID: c8e77a0d-22b5-4614-a5c9-d1761baff29c) for task 1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.832764 25241 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.832779 25242 slave.cpp:5879] Completing task 1 [17:58:36]W: [Step 10/10] I0324 17:58:36.832804 25242 slave.cpp:4143] Cleaning up executor '1' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 at executor(1)@172.30.2.190:54429 [17:58:36]W: [Step 10/10] I0324 17:58:36.832844 25241 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.833004 25243 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' for gc 6.99999035957333days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.833111 25243 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1' for gc 6.99999035822815days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.833206 25243 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' for gc 6.99999035763852days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.833269 25243 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1' for gc 6.99999035714074days in the future [17:58:36]W: [Step 10/10] W0324 17:58:36.837210 25241 status_update_manager.cpp:446] Acknowledged a terminal status update TASK_FAILED (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 but updates are still pending [17:58:36]W: [Step 10/10] I0324 17:58:36.837237 25241 status_update_manager.cpp:528] Cleaning up status update stream for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.837425 25245 slave.cpp:2548] Status update manager successfully handled status update acknowledgement (UUID: 0fa8bc21-7315-4d5f-bc59-9acf0eb3fe93) for task 0 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.837448 25241 status_update_manager.cpp:392] Received status update acknowledgement (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.837467 25245 slave.cpp:5879] Completing task 0 [17:58:36]W: [Step 10/10] I0324 17:58:36.837491 25245 slave.cpp:4143] Cleaning up executor '0' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 at executor(1)@172.30.2.190:47691 [17:58:36]W: [Step 10/10] I0324 17:58:36.837558 25241 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.837673 25247 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89' for gc 6.99999030550815days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.837749 25247 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0' for gc 6.99999030434074days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.837844 25247 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89' for gc 6.99999030375111days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.837908 25247 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0' for gc 6.99999030325037days in the future [17:58:36]W: [Step 10/10] I0324 17:58:36.841482 25244 slave.cpp:2548] Status update manager successfully handled status update acknowledgement (UUID: c69b2bb4-9439-4308-8ad8-f9099a36042f) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36] : [Step 10/10] Command exited with status 0 (pid: 29320) [17:58:36]W: [Step 10/10] I0324 17:58:36.905248 29315 exec.cpp:535] Executor sending status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.905758 25241 slave.cpp:3138] Handling status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from executor(1)@172.30.2.190:42013 [17:58:36]W: [Step 10/10] I0324 17:58:36.906630 25241 slave.cpp:5838] Terminating task 2 [17:58:36]W: [Step 10/10] I0324 17:58:36.907547 25246 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.907600 25246 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.911530 25246 status_update_manager.cpp:374] Forwarding update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to the slave [17:58:36]W: [Step 10/10] I0324 17:58:36.911734 25242 slave.cpp:3536] Forwarding the update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to master@172.30.2.190:33501 [17:58:36]W: [Step 10/10] I0324 17:58:36.911883 25242 slave.cpp:3430] Status update manager successfully handled status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.911926 25242 slave.cpp:3446] Sending acknowledgement for status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 to executor(1)@172.30.2.190:42013 [17:58:36]W: [Step 10/10] I0324 17:58:36.912005 25244 master.cpp:4947] Status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 from slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.912037 25244 master.cpp:4995] Forwarding status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.912163 25244 master.cpp:6608] Updating the state of task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (latest state: TASK_FINISHED, status update state: TASK_FINISHED) [17:58:36]W: [Step 10/10] I0324 17:58:36.912441 29316 exec.cpp:358] Executor received status update acknowledgement e9866622-a9ec-4d98-bb2f-9a8961c27035 for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.912489 25248 sched.cpp:982] Scheduler::statusUpdate took 209079ns [17:58:36]W: [Step 10/10] I0324 17:58:36.912482 25241 hierarchical.cpp:890] Recovered cpus(*):1; mem(*):1 (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: ) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.912674 25248 master.cpp:4102] Processing ACKNOWLEDGE call e9866622-a9ec-4d98-bb2f-9a8961c27035 for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:36]W: [Step 10/10] I0324 17:58:36.912734 25248 master.cpp:6674] Removing task 2 with resources cpus(*):1; mem(*):1 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 at slave(57)@172.30.2.190:33501 (ip-172-30-2-190.mesosphere.io) [17:58:36]W: [Step 10/10] I0324 17:58:36.913074 25248 status_update_manager.cpp:392] Received status update acknowledgement (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.913214 25248 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FINISHED (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.917573 25248 status_update_manager.cpp:528] Cleaning up status update stream for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.917774 25243 slave.cpp:2548] Status update manager successfully handled status update acknowledgement (UUID: e9866622-a9ec-4d98-bb2f-9a8961c27035) for task 2 of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:36]W: [Step 10/10] I0324 17:58:36.917814 25243 slave.cpp:5879] Completing task 2 [17:58:37]W: [Step 10/10] I0324 17:58:37.417965 25241 hierarchical.cpp:1582] No inverse offers to send out! [17:58:37]W: [Step 10/10] I0324 17:58:37.418033 25241 hierarchical.cpp:1138] Performed allocation for 1 slaves in 712889ns [17:58:37]W: [Step 10/10] I0324 17:58:37.418242 25242 master.cpp:5508] Sending 1 offers to framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:37]W: [Step 10/10] I0324 17:58:37.418645 25246 sched.cpp:874] Scheduler::resourceOffers took 103682ns [17:58:37]W: [Step 10/10] I0324 17:58:37.418902 25246 master.cpp:3825] Processing DECLINE call for offers: [ 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-O1 ] for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:37]W: [Step 10/10] I0324 17:58:37.419428 25244 hierarchical.cpp:890] Recovered cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: ) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:37]W: [Step 10/10] I0324 17:58:37.419471 25244 hierarchical.cpp:927] Framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 filtered slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for 5secs [17:58:37]W: [Step 10/10] I0324 17:58:37.907248 25245 slave.cpp:3664] executor(1)@172.30.2.190:42013 exited [17:58:37]W: [Step 10/10] I0324 17:58:37.942790 25241 containerizer.cpp:1656] Executor for container '86be5a57-b217-4535-8b84-d3709100f1a7' has exited [17:58:37]W: [Step 10/10] I0324 17:58:37.942821 25241 containerizer.cpp:1421] Destroying container '86be5a57-b217-4535-8b84-d3709100f1a7' [17:58:37]W: [Step 10/10] I0324 17:58:37.944432 25248 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/86be5a57-b217-4535-8b84-d3709100f1a7 [17:58:37]W: [Step 10/10] I0324 17:58:37.945746 25241 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/86be5a57-b217-4535-8b84-d3709100f1a7 after 1.268992ms [17:58:37]W: [Step 10/10] I0324 17:58:37.947329 25242 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/86be5a57-b217-4535-8b84-d3709100f1a7 [17:58:37]W: [Step 10/10] I0324 17:58:37.948657 25242 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/86be5a57-b217-4535-8b84-d3709100f1a7 after 1.287936ms [17:58:37]W: [Step 10/10] I0324 17:58:37.950976 25243 provisioner.cpp:306] Ignoring destroy request for unknown container 86be5a57-b217-4535-8b84-d3709100f1a7 [17:58:37]W: [Step 10/10] I0324 17:58:37.951222 25246 slave.cpp:4039] Executor '2' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 exited with status 0 [17:58:37]W: [Step 10/10] I0324 17:58:37.951282 25246 slave.cpp:4143] Cleaning up executor '2' of framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 at executor(1)@172.30.2.190:42013 [17:58:37]W: [Step 10/10] I0324 17:58:37.951535 25248 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' for gc 6.99998898768593days in the future [17:58:37]W: [Step 10/10] I0324 17:58:37.951624 25248 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2' for gc 6.99998898633185days in the future [17:58:37]W: [Step 10/10] I0324 17:58:37.951689 25248 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' for gc 6.99998898575111days in the future [17:58:37]W: [Step 10/10] I0324 17:58:37.951704 25246 slave.cpp:4231] Cleaning up framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:37]W: [Step 10/10] I0324 17:58:37.951756 25248 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2' for gc 6.99998898527704days in the future [17:58:37]W: [Step 10/10] I0324 17:58:37.951792 25244 status_update_manager.cpp:282] Closing status update streams for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:37]W: [Step 10/10] I0324 17:58:37.951831 25248 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000' for gc 6.99998898407407days in the future [17:58:37]W: [Step 10/10] I0324 17:58:37.951896 25248 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/meta/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000' for gc 6.99998898360889days in the future [17:58:38]W: [Step 10/10] I0324 17:58:38.419607 25241 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:38]W: [Step 10/10] I0324 17:58:38.419642 25241 hierarchical.cpp:1487] No resources available to allocate! [17:58:38]W: [Step 10/10] I0324 17:58:38.419670 25241 hierarchical.cpp:1582] No inverse offers to send out! [17:58:38]W: [Step 10/10] I0324 17:58:38.419716 25241 hierarchical.cpp:1138] Performed allocation for 1 slaves in 479161ns [17:58:39]W: [Step 10/10] I0324 17:58:39.421509 25245 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:39]W: [Step 10/10] I0324 17:58:39.421588 25245 hierarchical.cpp:1487] No resources available to allocate! [17:58:39]W: [Step 10/10] I0324 17:58:39.421629 25245 hierarchical.cpp:1582] No inverse offers to send out! [17:58:39]W: [Step 10/10] I0324 17:58:39.421702 25245 hierarchical.cpp:1138] Performed allocation for 1 slaves in 810178ns [17:58:40]W: [Step 10/10] I0324 17:58:40.423280 25246 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:40]W: [Step 10/10] I0324 17:58:40.423358 25246 hierarchical.cpp:1487] No resources available to allocate! [17:58:40]W: [Step 10/10] I0324 17:58:40.423398 25246 hierarchical.cpp:1582] No inverse offers to send out! [17:58:40]W: [Step 10/10] I0324 17:58:40.423472 25246 hierarchical.cpp:1138] Performed allocation for 1 slaves in 814475ns [17:58:41]W: [Step 10/10] I0324 17:58:41.425036 25241 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:41]W: [Step 10/10] I0324 17:58:41.425103 25241 hierarchical.cpp:1487] No resources available to allocate! [17:58:41]W: [Step 10/10] I0324 17:58:41.425139 25241 hierarchical.cpp:1582] No inverse offers to send out! [17:58:41]W: [Step 10/10] I0324 17:58:41.425205 25241 hierarchical.cpp:1138] Performed allocation for 1 slaves in 680657ns [17:58:42]W: [Step 10/10] I0324 17:58:42.426666 25246 hierarchical.cpp:1582] No inverse offers to send out! [17:58:42]W: [Step 10/10] I0324 17:58:42.426770 25246 hierarchical.cpp:1138] Performed allocation for 1 slaves in 870338ns [17:58:42]W: [Step 10/10] I0324 17:58:42.427001 25243 master.cpp:5508] Sending 1 offers to framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:42]W: [Step 10/10] I0324 17:58:42.427458 25248 sched.cpp:874] Scheduler::resourceOffers took 76952ns [17:58:42]W: [Step 10/10] I0324 17:58:42.427703 25248 master.cpp:3825] Processing DECLINE call for offers: [ 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-O2 ] for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:42]W: [Step 10/10] I0324 17:58:42.428207 25243 hierarchical.cpp:890] Recovered cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: ) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:42]W: [Step 10/10] I0324 17:58:42.428258 25243 hierarchical.cpp:927] Framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 filtered slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for 5secs [17:58:43]W: [Step 10/10] I0324 17:58:43.428006 25246 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:43]W: [Step 10/10] I0324 17:58:43.428066 25246 hierarchical.cpp:1487] No resources available to allocate! [17:58:43]W: [Step 10/10] I0324 17:58:43.428100 25246 hierarchical.cpp:1582] No inverse offers to send out! [17:58:43]W: [Step 10/10] I0324 17:58:43.428163 25246 hierarchical.cpp:1138] Performed allocation for 1 slaves in 643970ns [17:58:44]W: [Step 10/10] I0324 17:58:44.428881 25243 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:44]W: [Step 10/10] I0324 17:58:44.428953 25243 hierarchical.cpp:1487] No resources available to allocate! [17:58:44]W: [Step 10/10] I0324 17:58:44.428990 25243 hierarchical.cpp:1582] No inverse offers to send out! [17:58:44]W: [Step 10/10] I0324 17:58:44.429057 25243 hierarchical.cpp:1138] Performed allocation for 1 slaves in 721885ns [17:58:45]W: [Step 10/10] I0324 17:58:45.430439 25243 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:45]W: [Step 10/10] I0324 17:58:45.430518 25243 hierarchical.cpp:1487] No resources available to allocate! [17:58:45]W: [Step 10/10] I0324 17:58:45.430557 25243 hierarchical.cpp:1582] No inverse offers to send out! [17:58:45]W: [Step 10/10] I0324 17:58:45.430629 25243 hierarchical.cpp:1138] Performed allocation for 1 slaves in 782875ns [17:58:46]W: [Step 10/10] I0324 17:58:46.431428 25243 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:46]W: [Step 10/10] I0324 17:58:46.431502 25243 hierarchical.cpp:1487] No resources available to allocate! [17:58:46]W: [Step 10/10] I0324 17:58:46.431540 25243 hierarchical.cpp:1582] No inverse offers to send out! [17:58:46]W: [Step 10/10] I0324 17:58:46.431613 25243 hierarchical.cpp:1138] Performed allocation for 1 slaves in 803988ns [17:58:47]W: [Step 10/10] I0324 17:58:47.433408 25248 hierarchical.cpp:1582] No inverse offers to send out! [17:58:47]W: [Step 10/10] I0324 17:58:47.433519 25248 hierarchical.cpp:1138] Performed allocation for 1 slaves in 895766ns [17:58:47]W: [Step 10/10] I0324 17:58:47.433768 25244 master.cpp:5508] Sending 1 offers to framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:47]W: [Step 10/10] I0324 17:58:47.434211 25244 sched.cpp:874] Scheduler::resourceOffers took 75358ns [17:58:47]W: [Step 10/10] I0324 17:58:47.434463 25244 master.cpp:3825] Processing DECLINE call for offers: [ 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-O3 ] for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:47]W: [Step 10/10] I0324 17:58:47.434973 25243 hierarchical.cpp:890] Recovered cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] (total: cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000], allocated: ) on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 from framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:47]W: [Step 10/10] I0324 17:58:47.435029 25243 hierarchical.cpp:927] Framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 filtered slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for 5secs [17:58:48]W: [Step 10/10] I0324 17:58:48.434936 25245 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:48]W: [Step 10/10] I0324 17:58:48.434972 25245 hierarchical.cpp:1487] No resources available to allocate! [17:58:48]W: [Step 10/10] I0324 17:58:48.434999 25245 hierarchical.cpp:1582] No inverse offers to send out! [17:58:48]W: [Step 10/10] I0324 17:58:48.435044 25245 hierarchical.cpp:1138] Performed allocation for 1 slaves in 430708ns [17:58:49]W: [Step 10/10] I0324 17:58:49.436395 25242 hierarchical.cpp:1673] Filtered offer with cpus(*):1000; mem(*):1000; disk(*):3962; ports(*):[31000-32000] on slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:49]W: [Step 10/10] I0324 17:58:49.436473 25242 hierarchical.cpp:1487] No resources available to allocate! [17:58:49]W: [Step 10/10] I0324 17:58:49.436514 25242 hierarchical.cpp:1582] No inverse offers to send out! [17:58:49]W: [Step 10/10] I0324 17:58:49.436588 25242 hierarchical.cpp:1138] Performed allocation for 1 slaves in 808116ns [17:58:49]W: [Step 10/10] I0324 17:58:49.533778 25245 slave.cpp:4898] Querying resource estimator for oversubscribable resources [17:58:49]W: [Step 10/10] I0324 17:58:49.534081 25247 slave.cpp:4912] Received oversubscribable resources from the resource estimator [17:58:49]W: [Step 10/10] I0324 17:58:49.542191 25246 slave.cpp:3618] Received ping from slave-observer(55)@172.30.2.190:33501 [17:58:49] : [Step 10/10] ../../src/tests/fetcher_cache_tests.cpp:1141: Failure [17:58:49] : [Step 10/10] Failed to wait 15secs for awaitFinished(tasks.get()) [17:58:49]W: [Step 10/10] I0324 17:58:49.590600 25227 sched.cpp:1904] Asked to stop the driver [17:58:49]W: [Step 10/10] I0324 17:58:49.590695 25247 sched.cpp:1144] Stopping framework '9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000' [17:58:49] : [Step 10/10] Begin listing sandboxes [17:58:49] : [Step 10/10] Begin listing sandbox `/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/latest`: [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-cmd`: [17:58:49] : [Step 10/10] touch mesos-fetcher-test-cmd$1 [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `stdout`: [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-archive.tgz`: [17:58:49]W: [Step 10/10] I0324 17:58:49.590948 25248 master.cpp:6081] Processing TEARDOWN call for framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:49]W: [Step 10/10] I0324 17:58:49.590984 25248 master.cpp:6093] Removing framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 (default) at scheduler-f2e0b12c-147e-47aa-8736-24a03235ddb0@172.30.2.190:33501 [17:58:49]W: [Step 10/10] I0324 17:58:49.591145 25247 hierarchical.cpp:375] Deactivated framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:49]W: [Step 10/10] I0324 17:58:49.591219 25244 slave.cpp:2215] Asked to shut down framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 by master@172.30.2.190:33501 [17:58:49]W: [Step 10/10] W0324 17:58:49.591249 25244 slave.cpp:2230] Cannot shut down unknown framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:49] : [Step 10/10] mesos-fetcher-test-acmd 0000777 0000000 0000000 00000000037 12675025312 014134 0 ustar root root touch mesos-fetcher-test-acmd$1 [17:58:49]W: [Step 10/10] I0324 17:58:49.591423 25248 master.cpp:1083] Master terminating [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `stderr`: [17:58:49]W: [Step 10/10] I0324 17:58:49.591485 25247 hierarchical.cpp:326] Removed framework 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000 [17:58:49] : [Step 10/10] I0324 17:58:34.644672 29192 logging.cpp:195] Logging to STDERR [17:58:49]W: [Step 10/10] I0324 17:58:49.591853 25247 hierarchical.cpp:505] Removed slave 9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0 [17:58:49] : [Step 10/10] I0324 17:58:34.644773 29192 fetcher.cpp:430] Fetcher Info: {"cache_directory":"\/tmp\/FetcherCacheHttpTest_HttpMixed_iZPPj9\/fetch\/slaves\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0\/root","items":[{"action":"DOWNLOAD_AND_CACHE","cache_filename":"c1-mesos-fetc_rchive.tgz","uri":{"cache":true,"executable":false,"extract":false,"value":"http:\/\/172.30.2.190:33501\/(2689)\/mesos-fetcher-test-archive.tgz"}},{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":false,"value":"http:\/\/172.30.2.190:33501\/(2689)\/mesos-fetcher-test-cmd"}}],"sandbox_directory":"\/tmp\/FetcherCacheHttpTest_HttpMixed_iZPPj9\/slaves\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0\/frameworks\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000\/executors\/0\/runs\/892d0f4a-6e99-470a-a85b-52e2bbcb7e89","user":"root"} [17:58:49]W: [Step 10/10] I0324 17:58:49.592092 25247 slave.cpp:3664] master@172.30.2.190:33501 exited [17:58:49] : [Step 10/10] I0324 17:58:34.650003 29192 fetcher.cpp:385] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' [17:58:49]W: [Step 10/10] W0324 17:58:49.592118 25247 slave.cpp:3667] Master disconnected! Waiting for a new master to be elected [17:58:49] : [Step 10/10] I0324 17:58:34.650024 29192 fetcher.cpp:355] Downloading into cache [17:58:49] : [Step 10/10] I0324 17:58:34.650141 29192 fetcher.cpp:187] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.650176 29192 fetcher.cpp:134] Downloading resource from 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/fetch/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/root/c1-mesos-fetc_rchive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.652191 29192 fetcher.cpp:292] Fetching from cache [17:58:49] : [Step 10/10] I0324 17:58:34.652228 29192 fetcher.cpp:167] Copying resource with command:cp '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/fetch/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/root/c1-mesos-fetc_rchive.tgz' '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.654259 29192 fetcher.cpp:462] Fetched 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.654284 29192 fetcher.cpp:385] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.654291 29192 fetcher.cpp:250] Fetching directly into the sandbox directory [17:58:49] : [Step 10/10] I0324 17:58:34.654310 29192 fetcher.cpp:187] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.654327 29192 fetcher.cpp:134] Downloading resource from 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.655557 29192 fetcher.cpp:462] Fetched 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/0/runs/892d0f4a-6e99-470a-a85b-52e2bbcb7e89/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-cmd0`: [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] End sandbox [17:58:49] : [Step 10/10] Begin listing sandbox `/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/latest`: [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-cmd`: [17:58:49] : [Step 10/10] touch mesos-fetcher-test-cmd$1 [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-acmd1`: [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `stdout`: [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-archive.tgz`: [17:58:49] : [Step 10/10] mesos-fetcher-test-acmd 0000777 0000000 0000000 00000000037 12675025312 014134 0 ustar root root touch mesos-fetcher-test-acmd$1 [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-acmd`: [17:58:49] : [Step 10/10] touch mesos-fetcher-test-acmd$1 [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `stderr`: [17:58:49] : [Step 10/10] I0324 17:58:34.643620 29185 logging.cpp:195] Logging to STDERR [17:58:49] : [Step 10/10] I0324 17:58:34.643781 29185 fetcher.cpp:430] Fetcher Info: {"cache_directory":"\/tmp\/FetcherCacheHttpTest_HttpMixed_iZPPj9\/fetch\/slaves\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0\/root","items":[{"action":"BYPASS_CACHE","uri":{"executable":false,"extract":true,"value":"http:\/\/172.30.2.190:33501\/(2689)\/mesos-fetcher-test-archive.tgz"}},{"action":"BYPASS_CACHE","uri":{"executable":false,"extract":true,"value":"http:\/\/172.30.2.190:33501\/(2689)\/mesos-fetcher-test-cmd"}}],"sandbox_directory":"\/tmp\/FetcherCacheHttpTest_HttpMixed_iZPPj9\/slaves\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0\/frameworks\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000\/executors\/1\/runs\/bd979c82-be7b-48b4-931b-26c848e8b592","user":"root"} [17:58:49] : [Step 10/10] I0324 17:58:34.649360 29185 fetcher.cpp:385] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.649386 29185 fetcher.cpp:250] Fetching directly into the sandbox directory [17:58:49]W: [Step 10/10] I0324 17:58:49.596443 25243 slave.cpp:800] Slave terminating [17:58:49] : [Step 10/10] I0324 17:58:34.649423 29185 fetcher.cpp:187] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.649462 29185 fetcher.cpp:134] Downloading resource from 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.651794 29185 fetcher.cpp:84] Extracting with command: tar -C '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' -xf '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.654259 29185 fetcher.cpp:92] Extracted '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/mesos-fetcher-test-archive.tgz' into '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592' [17:58:49] : [Step 10/10] I0324 17:58:34.654287 29185 fetcher.cpp:462] Fetched 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.654299 29185 fetcher.cpp:385] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.654306 29185 fetcher.cpp:250] Fetching directly into the sandbox directory [17:58:49] : [Step 10/10] I0324 17:58:34.654325 29185 fetcher.cpp:187] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.654341 29185 fetcher.cpp:134] Downloading resource from 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] W0324 17:58:34.655880 29185 fetcher.cpp:275] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd [17:58:49] : [Step 10/10] I0324 17:58:34.655910 29185 fetcher.cpp:462] Fetched 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/1/runs/bd979c82-be7b-48b4-931b-26c848e8b592/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] End sandbox [17:58:49] : [Step 10/10] Begin listing sandbox `/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/latest`: [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-cmd`: [17:58:49] : [Step 10/10] touch mesos-fetcher-test-cmd$1 [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `stdout`: [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-acmd2`: [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `mesos-fetcher-test-acmd`: [17:58:49] : [Step 10/10] touch mesos-fetcher-test-acmd$1 [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] Begin file contents of `stderr`: [17:58:49] : [Step 10/10] I0324 17:58:34.771075 29239 logging.cpp:195] Logging to STDERR [17:58:49] : [Step 10/10] I0324 17:58:34.771231 29239 fetcher.cpp:430] Fetcher Info: {"cache_directory":"\/tmp\/FetcherCacheHttpTest_HttpMixed_iZPPj9\/fetch\/slaves\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0\/root","items":[{"action":"RETRIEVE_FROM_CACHE","cache_filename":"c1-mesos-fetc_rchive.tgz","uri":{"cache":true,"executable":false,"extract":true,"value":"http:\/\/172.30.2.190:33501\/(2689)\/mesos-fetcher-test-archive.tgz"}},{"action":"BYPASS_CACHE","uri":{"executable":false,"extract":false,"value":"http:\/\/172.30.2.190:33501\/(2689)\/mesos-fetcher-test-cmd"}}],"sandbox_directory":"\/tmp\/FetcherCacheHttpTest_HttpMixed_iZPPj9\/slaves\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0\/frameworks\/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000\/executors\/2\/runs\/86be5a57-b217-4535-8b84-d3709100f1a7","user":"root"} [17:58:49] : [Step 10/10] I0324 17:58:34.776996 29239 fetcher.cpp:385] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.777019 29239 fetcher.cpp:292] Fetching from cache [17:58:49] : [Step 10/10] I0324 17:58:34.777047 29239 fetcher.cpp:84] Extracting with command: tar -C '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' -xf '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/fetch/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/root/c1-mesos-fetc_rchive.tgz' [17:58:49] : [Step 10/10] I0324 17:58:34.779330 29239 fetcher.cpp:92] Extracted '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/fetch/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/root/c1-mesos-fetc_rchive.tgz' into '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' [17:58:49] : [Step 10/10] I0324 17:58:34.779361 29239 fetcher.cpp:462] Fetched 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-archive.tgz' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7' [17:58:49] : [Step 10/10] I0324 17:58:34.779371 29239 fetcher.cpp:385] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.779377 29239 fetcher.cpp:250] Fetching directly into the sandbox directory [17:58:49] : [Step 10/10] I0324 17:58:34.779400 29239 fetcher.cpp:187] Fetching URI 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.779440 29239 fetcher.cpp:134] Downloading resource from 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] I0324 17:58:34.782606 29239 fetcher.cpp:462] Fetched 'http://172.30.2.190:33501/(2689)/mesos-fetcher-test-cmd' to '/tmp/FetcherCacheHttpTest_HttpMixed_iZPPj9/slaves/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-S0/frameworks/9a1809d1-13d2-41fb-bc50-5fcd03a1082d-0000/executors/2/runs/86be5a57-b217-4535-8b84-d3709100f1a7/mesos-fetcher-test-cmd' [17:58:49] : [Step 10/10] [17:58:49] : [Step 10/10] End file [17:58:49] : [Step 10/10] End sandbox [17:58:49] : [Step 10/10] End sandboxes [17:58:49] : [Step 10/10] [ FAILED ] FetcherCacheHttpTest.HttpMixed (15215 ms)
From a known good run
[12:21:23] : [Step 11/11] [ RUN ] FetcherCacheHttpTest.HttpMixed [12:21:23]W: [Step 11/11] I0324 12:21:23.828660 1814 cluster.cpp:139] Creating default 'local' authorizer [12:21:23]W: [Step 11/11] I0324 12:21:23.832818 1814 leveldb.cpp:174] Opened db in 3.964758ms [12:21:23]W: [Step 11/11] I0324 12:21:23.834213 1814 leveldb.cpp:181] Compacted db in 1.36947ms [12:21:23]W: [Step 11/11] I0324 12:21:23.834254 1814 leveldb.cpp:196] Created db iterator in 15905ns [12:21:23]W: [Step 11/11] I0324 12:21:23.834265 1814 leveldb.cpp:202] Seeked to beginning of db in 1381ns [12:21:23]W: [Step 11/11] I0324 12:21:23.834272 1814 leveldb.cpp:271] Iterated through 0 keys in the db in 376ns [12:21:23]W: [Step 11/11] I0324 12:21:23.834306 1814 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned [12:21:23]W: [Step 11/11] I0324 12:21:23.834720 1830 recover.cpp:447] Starting replica recovery [12:21:23]W: [Step 11/11] I0324 12:21:23.834955 1830 recover.cpp:473] Replica is in EMPTY status [12:21:23]W: [Step 11/11] I0324 12:21:23.835834 1834 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from (2690)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.836252 1833 recover.cpp:193] Received a recover response from a replica in EMPTY status [12:21:23]W: [Step 11/11] I0324 12:21:23.836784 1832 recover.cpp:564] Updating replica status to STARTING [12:21:23]W: [Step 11/11] I0324 12:21:23.837311 1829 master.cpp:376] Master b8970495-be32-4130-94ae-464e22ce731b (ip-172-30-2-102.mesosphere.io) started on 172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.837329 1829 master.cpp:378] Flags at startup: --acls="" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="true" --authenticate_http="true" --authenticate_slaves="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/DmDlcT/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/DmDlcT/master" --zk_session_timeout="10secs" [12:21:23]W: [Step 11/11] I0324 12:21:23.837560 1829 master.cpp:427] Master only allowing authenticated frameworks to register [12:21:23]W: [Step 11/11] I0324 12:21:23.837569 1829 master.cpp:432] Master only allowing authenticated slaves to register [12:21:23]W: [Step 11/11] I0324 12:21:23.837575 1829 credentials.hpp:35] Loading credentials for authentication from '/tmp/DmDlcT/credentials' [12:21:23]W: [Step 11/11] I0324 12:21:23.837797 1829 master.cpp:474] Using default 'crammd5' authenticator [12:21:23]W: [Step 11/11] I0324 12:21:23.837910 1829 master.cpp:545] Using default 'basic' HTTP authenticator [12:21:23]W: [Step 11/11] I0324 12:21:23.838019 1835 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.024114ms [12:21:23]W: [Step 11/11] I0324 12:21:23.838045 1835 replica.cpp:320] Persisted replica status to STARTING [12:21:23]W: [Step 11/11] I0324 12:21:23.838054 1829 master.cpp:583] Authorization enabled [12:21:23]W: [Step 11/11] I0324 12:21:23.838201 1830 hierarchical.cpp:144] Initialized hierarchical allocator process [12:21:23]W: [Step 11/11] I0324 12:21:23.838234 1828 whitelist_watcher.cpp:77] No whitelist given [12:21:23]W: [Step 11/11] I0324 12:21:23.838238 1832 recover.cpp:473] Replica is in STARTING status [12:21:23]W: [Step 11/11] I0324 12:21:23.839169 1828 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from (2692)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.839504 1831 recover.cpp:193] Received a recover response from a replica in STARTING status [12:21:23]W: [Step 11/11] I0324 12:21:23.839963 1834 recover.cpp:564] Updating replica status to VOTING [12:21:23]W: [Step 11/11] I0324 12:21:23.840394 1832 master.cpp:1826] The newly elected leader is master@172.30.2.102:40940 with id b8970495-be32-4130-94ae-464e22ce731b [12:21:23]W: [Step 11/11] I0324 12:21:23.840425 1832 master.cpp:1839] Elected as the leading master! [12:21:23]W: [Step 11/11] I0324 12:21:23.840447 1832 master.cpp:1526] Recovering from registrar [12:21:23]W: [Step 11/11] I0324 12:21:23.840602 1833 registrar.cpp:307] Recovering registrar [12:21:23]W: [Step 11/11] I0324 12:21:23.841173 1828 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 1.017803ms [12:21:23]W: [Step 11/11] I0324 12:21:23.841197 1828 replica.cpp:320] Persisted replica status to VOTING [12:21:23]W: [Step 11/11] I0324 12:21:23.841282 1835 recover.cpp:578] Successfully joined the Paxos group [12:21:23]W: [Step 11/11] I0324 12:21:23.841423 1835 recover.cpp:462] Recover process terminated [12:21:23]W: [Step 11/11] I0324 12:21:23.841809 1833 log.cpp:659] Attempting to start the writer [12:21:23]W: [Step 11/11] I0324 12:21:23.842792 1829 replica.cpp:493] Replica received implicit promise request from (2693)@172.30.2.102:40940 with proposal 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.843752 1829 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 934402ns [12:21:23]W: [Step 11/11] I0324 12:21:23.843770 1829 replica.cpp:342] Persisted promised to 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.844219 1828 coordinator.cpp:238] Coordinator attempting to fill missing positions [12:21:23]W: [Step 11/11] I0324 12:21:23.845257 1833 replica.cpp:388] Replica received explicit promise request from (2694)@172.30.2.102:40940 for position 0 with proposal 2 [12:21:23]W: [Step 11/11] I0324 12:21:23.846673 1833 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 1.383446ms [12:21:23]W: [Step 11/11] I0324 12:21:23.846695 1833 replica.cpp:712] Persisted action at 0 [12:21:23]W: [Step 11/11] I0324 12:21:23.847532 1830 replica.cpp:537] Replica received write request for position 0 from (2695)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.847582 1830 leveldb.cpp:436] Reading position from leveldb took 22346ns [12:21:23]W: [Step 11/11] I0324 12:21:23.848500 1830 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 885030ns [12:21:23]W: [Step 11/11] I0324 12:21:23.848520 1830 replica.cpp:712] Persisted action at 0 [12:21:23]W: [Step 11/11] I0324 12:21:23.849037 1828 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 [12:21:23]W: [Step 11/11] I0324 12:21:23.849992 1828 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 930735ns [12:21:23]W: [Step 11/11] I0324 12:21:23.850011 1828 replica.cpp:712] Persisted action at 0 [12:21:23]W: [Step 11/11] I0324 12:21:23.850025 1828 replica.cpp:697] Replica learned NOP action at position 0 [12:21:23]W: [Step 11/11] I0324 12:21:23.850440 1834 log.cpp:675] Writer started with ending position 0 [12:21:23]W: [Step 11/11] I0324 12:21:23.851353 1835 leveldb.cpp:436] Reading position from leveldb took 23375ns [12:21:23]W: [Step 11/11] I0324 12:21:23.852097 1835 registrar.cpp:340] Successfully fetched the registry (0B) in 11.457024ms [12:21:23]W: [Step 11/11] I0324 12:21:23.852193 1835 registrar.cpp:439] Applied 1 operations in 24431ns; attempting to update the 'registry' [12:21:23]W: [Step 11/11] I0324 12:21:23.852826 1829 log.cpp:683] Attempting to append 210 bytes to the log [12:21:23]W: [Step 11/11] I0324 12:21:23.852931 1830 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.853534 1834 replica.cpp:537] Replica received write request for position 1 from (2696)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.854466 1834 leveldb.cpp:341] Persisting action (229 bytes) to leveldb took 900259ns [12:21:23]W: [Step 11/11] I0324 12:21:23.854485 1834 replica.cpp:712] Persisted action at 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.855011 1833 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 [12:21:23]W: [Step 11/11] I0324 12:21:23.856021 1833 leveldb.cpp:341] Persisting action (231 bytes) to leveldb took 986034ns [12:21:23]W: [Step 11/11] I0324 12:21:23.856040 1833 replica.cpp:712] Persisted action at 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.856055 1833 replica.cpp:697] Replica learned APPEND action at position 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.856855 1835 registrar.cpp:484] Successfully updated the 'registry' in 4.602112ms [12:21:23]W: [Step 11/11] I0324 12:21:23.856959 1835 registrar.cpp:370] Successfully recovered registrar [12:21:23]W: [Step 11/11] I0324 12:21:23.857028 1833 log.cpp:702] Attempting to truncate the log to 1 [12:21:23]W: [Step 11/11] I0324 12:21:23.857151 1828 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 [12:21:23]W: [Step 11/11] I0324 12:21:23.857434 1829 master.cpp:1634] Recovered 0 slaves from the Registry (171B) ; allowing 10mins for slaves to re-register [12:21:23]W: [Step 11/11] I0324 12:21:23.857486 1834 hierarchical.cpp:171] Skipping recovery of hierarchical allocator: nothing to recover [12:21:23]W: [Step 11/11] I0324 12:21:23.857921 1832 replica.cpp:537] Replica received write request for position 2 from (2697)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.858886 1832 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 936049ns [12:21:23]W: [Step 11/11] I0324 12:21:23.858904 1832 replica.cpp:712] Persisted action at 2 [12:21:23]W: [Step 11/11] I0324 12:21:23.859439 1831 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 [12:21:23]W: [Step 11/11] I0324 12:21:23.860400 1831 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 936974ns [12:21:23]W: [Step 11/11] I0324 12:21:23.860446 1831 leveldb.cpp:399] Deleting ~1 keys from leveldb took 26495ns [12:21:23]W: [Step 11/11] I0324 12:21:23.860461 1831 replica.cpp:712] Persisted action at 2 [12:21:23]W: [Step 11/11] I0324 12:21:23.860476 1831 replica.cpp:697] Replica learned TRUNCATE action at position 2 [12:21:23]W: [Step 11/11] I0324 12:21:23.868588 1814 containerizer.cpp:149] Using isolation: posix/cpu,posix/mem,filesystem/posix [12:21:23]W: [Step 11/11] I0324 12:21:23.875233 1814 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher [12:21:23]W: [Step 11/11] I0324 12:21:23.878830 1830 slave.cpp:199] Slave started on 57)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.878851 1830 slave.cpp:200] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="true" --hostname_lookup="true" --http_authenticators="basic" --http_credentials="/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:1000;mem:1000" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C" [12:21:23]W: [Step 11/11] I0324 12:21:23.879197 1830 credentials.hpp:83] Loading credential for authentication from '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/credential' [12:21:23]W: [Step 11/11] I0324 12:21:23.879324 1830 slave.cpp:337] Slave using credential for: test-principal [12:21:23]W: [Step 11/11] I0324 12:21:23.879359 1830 credentials.hpp:35] Loading credentials for authentication from '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/http_credentials' [12:21:23]W: [Step 11/11] I0324 12:21:23.879520 1830 slave.cpp:389] Using default 'basic' HTTP authenticator [12:21:23]W: [Step 11/11] I0324 12:21:23.879762 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:1000;mem:1000 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.880115 1830 slave.cpp:588] Slave resources: cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000] [12:21:23]W: [Step 11/11] I0324 12:21:23.880168 1830 slave.cpp:596] Slave attributes: [ ] [12:21:23]W: [Step 11/11] I0324 12:21:23.880178 1830 slave.cpp:601] Slave hostname: ip-172-30-2-102.mesosphere.io [12:21:23]W: [Step 11/11] I0324 12:21:23.881074 1829 state.cpp:58] Recovering state from '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta' [12:21:23]W: [Step 11/11] I0324 12:21:23.881289 1832 status_update_manager.cpp:200] Recovering status update manager [12:21:23]W: [Step 11/11] I0324 12:21:23.881506 1830 containerizer.cpp:407] Recovering containerizer [12:21:23]W: [Step 11/11] I0324 12:21:23.883121 1830 provisioner.cpp:245] Provisioner recovery complete [12:21:23]W: [Step 11/11] I0324 12:21:23.883424 1828 slave.cpp:4700] Finished recovery [12:21:23]W: [Step 11/11] I0324 12:21:23.883726 1828 slave.cpp:4872] Querying resource estimator for oversubscribable resources [12:21:23]W: [Step 11/11] I0324 12:21:23.883909 1828 slave.cpp:927] New master detected at master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.883919 1831 status_update_manager.cpp:174] Pausing sending status updates [12:21:23]W: [Step 11/11] I0324 12:21:23.884006 1828 slave.cpp:990] Authenticating with master master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.884022 1828 slave.cpp:995] Using default CRAM-MD5 authenticatee [12:21:23]W: [Step 11/11] I0324 12:21:23.884131 1828 slave.cpp:963] Detecting new master [12:21:23]W: [Step 11/11] I0324 12:21:23.884186 1829 authenticatee.cpp:121] Creating new client SASL connection [12:21:23]W: [Step 11/11] I0324 12:21:23.884240 1828 slave.cpp:4886] Received oversubscribable resources from the resource estimator [12:21:23]W: [Step 11/11] I0324 12:21:23.884475 1833 master.cpp:5679] Authenticating slave(57)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.884564 1831 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(159)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.884747 1830 authenticator.cpp:98] Creating new server SASL connection [12:21:23]W: [Step 11/11] I0324 12:21:23.884980 1829 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 [12:21:23]W: [Step 11/11] I0324 12:21:23.885002 1829 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' [12:21:23]W: [Step 11/11] I0324 12:21:23.885099 1829 authenticator.cpp:203] Received SASL authentication start [12:21:23]W: [Step 11/11] I0324 12:21:23.885133 1829 authenticator.cpp:325] Authentication requires more steps [12:21:23]W: [Step 11/11] I0324 12:21:23.885213 1829 authenticatee.cpp:258] Received SASL authentication step [12:21:23]W: [Step 11/11] I0324 12:21:23.885311 1829 authenticator.cpp:231] Received SASL authentication step [12:21:23]W: [Step 11/11] I0324 12:21:23.885334 1829 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-102.mesosphere.io' server FQDN: 'ip-172-30-2-102.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [12:21:23]W: [Step 11/11] I0324 12:21:23.885342 1829 auxprop.cpp:179] Looking up auxiliary property '*userPassword' [12:21:23]W: [Step 11/11] I0324 12:21:23.885370 1829 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [12:21:23]W: [Step 11/11] I0324 12:21:23.885388 1829 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-102.mesosphere.io' server FQDN: 'ip-172-30-2-102.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [12:21:23]W: [Step 11/11] I0324 12:21:23.885397 1829 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [12:21:23]W: [Step 11/11] I0324 12:21:23.885402 1829 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [12:21:23]W: [Step 11/11] I0324 12:21:23.885416 1829 authenticator.cpp:317] Authentication success [12:21:23]W: [Step 11/11] I0324 12:21:23.885489 1831 authenticatee.cpp:298] Authentication success [12:21:23]W: [Step 11/11] I0324 12:21:23.885530 1835 master.cpp:5709] Successfully authenticated principal 'test-principal' at slave(57)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.885603 1830 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(159)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.885835 1833 slave.cpp:1060] Successfully authenticated with master master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.885933 1833 slave.cpp:1456] Will retry registration in 16.231199ms if necessary [12:21:23]W: [Step 11/11] I0324 12:21:23.886106 1828 master.cpp:4390] Registering slave at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) with id b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:23]W: [Step 11/11] I0324 12:21:23.886485 1831 registrar.cpp:439] Applied 1 operations in 45621ns; attempting to update the 'registry' [12:21:23]W: [Step 11/11] I0324 12:21:23.887089 1832 log.cpp:683] Attempting to append 396 bytes to the log [12:21:23]W: [Step 11/11] I0324 12:21:23.887198 1829 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 [12:21:23]W: [Step 11/11] I0324 12:21:23.887796 1828 replica.cpp:537] Replica received write request for position 3 from (2715)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.889204 1828 leveldb.cpp:341] Persisting action (415 bytes) to leveldb took 1.37713ms [12:21:23]W: [Step 11/11] I0324 12:21:23.889227 1828 replica.cpp:712] Persisted action at 3 [12:21:23]W: [Step 11/11] I0324 12:21:23.889771 1835 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 [12:21:23]W: [Step 11/11] I0324 12:21:23.891773 1835 leveldb.cpp:341] Persisting action (417 bytes) to leveldb took 1.978624ms [12:21:23]W: [Step 11/11] I0324 12:21:23.891798 1835 replica.cpp:712] Persisted action at 3 [12:21:23]W: [Step 11/11] I0324 12:21:23.891811 1835 replica.cpp:697] Replica learned APPEND action at position 3 [12:21:23]W: [Step 11/11] I0324 12:21:23.893044 1835 registrar.cpp:484] Successfully updated the 'registry' in 6.5152ms [12:21:23]W: [Step 11/11] I0324 12:21:23.893252 1834 log.cpp:702] Attempting to truncate the log to 3 [12:21:23]W: [Step 11/11] I0324 12:21:23.893353 1831 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 [12:21:23]W: [Step 11/11] I0324 12:21:23.893626 1829 slave.cpp:3617] Received ping from slave-observer(56)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.893755 1830 master.cpp:4458] Registered slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) with cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000] [12:21:23]W: [Step 11/11] I0324 12:21:23.893945 1828 hierarchical.cpp:473] Added slave b8970495-be32-4130-94ae-464e22ce731b-S0 (ip-172-30-2-102.mesosphere.io) with cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000] (allocated: ) [12:21:23]W: [Step 11/11] I0324 12:21:23.894006 1832 slave.cpp:1104] Registered with master master@172.30.2.102:40940; given slave ID b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:23]W: [Step 11/11] I0324 12:21:23.894033 1832 fetcher.cpp:81] Clearing fetcher cache [12:21:23]W: [Step 11/11] I0324 12:21:23.894109 1828 hierarchical.cpp:1466] No resources available to allocate! [12:21:23]W: [Step 11/11] I0324 12:21:23.894127 1829 replica.cpp:537] Replica received write request for position 4 from (2716)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.894165 1828 hierarchical.cpp:1161] Performed allocation for slave b8970495-be32-4130-94ae-464e22ce731b-S0 in 176110ns [12:21:23]W: [Step 11/11] I0324 12:21:23.894220 1828 status_update_manager.cpp:181] Resuming sending status updates [12:21:23]W: [Step 11/11] I0324 12:21:23.894387 1832 slave.cpp:1127] Checkpointing SlaveInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/slave.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.894647 1832 slave.cpp:1164] Forwarding total oversubscribed resources [12:21:23]W: [Step 11/11] I0324 12:21:23.894798 1832 master.cpp:4802] Received update of slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) with total oversubscribed resources [12:21:23]W: [Step 11/11] I0324 12:21:23.894837 1814 sched.cpp:222] Version: 0.29.0 [12:21:23]W: [Step 11/11] I0324 12:21:23.895145 1829 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 980946ns [12:21:23]W: [Step 11/11] I0324 12:21:23.895172 1829 replica.cpp:712] Persisted action at 4 [12:21:23]W: [Step 11/11] I0324 12:21:23.895341 1835 sched.cpp:326] New master detected at master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.895416 1835 sched.cpp:382] Authenticating with master master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.895431 1835 sched.cpp:389] Using default CRAM-MD5 authenticatee [12:21:23]W: [Step 11/11] I0324 12:21:23.895449 1831 hierarchical.cpp:531] Slave b8970495-be32-4130-94ae-464e22ce731b-S0 (ip-172-30-2-102.mesosphere.io) updated with oversubscribed resources (total: cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000], allocated: ) [12:21:23]W: [Step 11/11] I0324 12:21:23.895552 1831 hierarchical.cpp:1466] No resources available to allocate! [12:21:23]W: [Step 11/11] I0324 12:21:23.895596 1831 hierarchical.cpp:1161] Performed allocation for slave b8970495-be32-4130-94ae-464e22ce731b-S0 in 116282ns [12:21:23]W: [Step 11/11] I0324 12:21:23.895653 1829 authenticatee.cpp:121] Creating new client SASL connection [12:21:23]W: [Step 11/11] I0324 12:21:23.895808 1832 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 [12:21:23]W: [Step 11/11] I0324 12:21:23.895901 1831 master.cpp:5679] Authenticating scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.896001 1830 authenticator.cpp:413] Starting authentication session for crammd5_authenticatee(160)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.896193 1835 authenticator.cpp:98] Creating new server SASL connection [12:21:23]W: [Step 11/11] I0324 12:21:23.896420 1829 authenticatee.cpp:212] Received SASL authentication mechanisms: CRAM-MD5 [12:21:23]W: [Step 11/11] I0324 12:21:23.896440 1829 authenticatee.cpp:238] Attempting to authenticate with mechanism 'CRAM-MD5' [12:21:23]W: [Step 11/11] I0324 12:21:23.896519 1829 authenticator.cpp:203] Received SASL authentication start [12:21:23]W: [Step 11/11] I0324 12:21:23.896553 1829 authenticator.cpp:325] Authentication requires more steps [12:21:23]W: [Step 11/11] I0324 12:21:23.896630 1829 authenticatee.cpp:258] Received SASL authentication step [12:21:23]W: [Step 11/11] I0324 12:21:23.896713 1830 authenticator.cpp:231] Received SASL authentication step [12:21:23]W: [Step 11/11] I0324 12:21:23.896744 1830 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-102.mesosphere.io' server FQDN: 'ip-172-30-2-102.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [12:21:23]W: [Step 11/11] I0324 12:21:23.896756 1830 auxprop.cpp:179] Looking up auxiliary property '*userPassword' [12:21:23]W: [Step 11/11] I0324 12:21:23.896781 1830 auxprop.cpp:179] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [12:21:23]W: [Step 11/11] I0324 12:21:23.896800 1830 auxprop.cpp:107] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-102.mesosphere.io' server FQDN: 'ip-172-30-2-102.mesosphere.io' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [12:21:23]W: [Step 11/11] I0324 12:21:23.896808 1830 auxprop.cpp:129] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [12:21:23]W: [Step 11/11] I0324 12:21:23.896813 1830 auxprop.cpp:129] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [12:21:23]W: [Step 11/11] I0324 12:21:23.896824 1830 authenticator.cpp:317] Authentication success [12:21:23]W: [Step 11/11] I0324 12:21:23.896898 1829 authenticatee.cpp:298] Authentication success [12:21:23]W: [Step 11/11] I0324 12:21:23.896931 1828 master.cpp:5709] Successfully authenticated principal 'test-principal' at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.896965 1834 authenticator.cpp:431] Authentication session cleanup for crammd5_authenticatee(160)@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.897171 1831 sched.cpp:472] Successfully authenticated with master master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.897195 1831 sched.cpp:777] Sending SUBSCRIBE call to master@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.897253 1832 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 1.419003ms [12:21:23]W: [Step 11/11] I0324 12:21:23.897272 1831 sched.cpp:810] Will retry registration in 1.211654712secs if necessary [12:21:23]W: [Step 11/11] I0324 12:21:23.897313 1832 leveldb.cpp:399] Deleting ~2 keys from leveldb took 35510ns [12:21:23]W: [Step 11/11] I0324 12:21:23.897337 1832 replica.cpp:712] Persisted action at 4 [12:21:23]W: [Step 11/11] I0324 12:21:23.897351 1828 master.cpp:2346] Received SUBSCRIBE call for framework 'default' at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.897354 1832 replica.cpp:697] Replica learned TRUNCATE action at position 4 [12:21:23]W: [Step 11/11] W0324 12:21:23.897377 1828 master.cpp:2353] Framework at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 (authenticated as 'test-principal') does not set 'principal' in FrameworkInfo [12:21:23]W: [Step 11/11] I0324 12:21:23.897414 1828 master.cpp:1865] Authorizing framework principal '' to receive offers for role '*' [12:21:23]W: [Step 11/11] I0324 12:21:23.897644 1833 master.cpp:2417] Subscribing framework default with checkpointing enabled and capabilities [ ] [12:21:23]W: [Step 11/11] I0324 12:21:23.897989 1828 hierarchical.cpp:265] Added framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.898141 1829 sched.cpp:704] Framework registered with b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.898185 1829 sched.cpp:718] Scheduler::registered took 19719ns [12:21:23]W: [Step 11/11] I0324 12:21:23.898591 1828 hierarchical.cpp:1561] No inverse offers to send out! [12:21:23]W: [Step 11/11] I0324 12:21:23.898656 1828 hierarchical.cpp:1138] Performed allocation for 1 slaves in 625820ns [12:21:23]W: [Step 11/11] I0324 12:21:23.899039 1833 master.cpp:5508] Sending 1 offers to framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.899413 1835 sched.cpp:874] Scheduler::resourceOffers took 93535ns [12:21:23]W: [Step 11/11] I0324 12:21:23.899766 1814 resources.cpp:572] Parsing resources as JSON failed: cpus:1;mem:1 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.899932 1814 resources.cpp:572] Parsing resources as JSON failed: cpus:1;mem:1 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.900109 1814 resources.cpp:572] Parsing resources as JSON failed: cpus:1;mem:1 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.901312 1832 master.cpp:3288] Processing ACCEPT call for offers: [ b8970495-be32-4130-94ae-464e22ce731b-O0 ] on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) for framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:23]W: [Step 11/11] I0324 12:21:23.901348 1832 master.cpp:2891] Authorizing framework principal '' to launch task 0 as user 'root' [12:21:23]W: [Step 11/11] I0324 12:21:23.901443 1832 master.cpp:2891] Authorizing framework principal '' to launch task 1 as user 'root' [12:21:23]W: [Step 11/11] I0324 12:21:23.901535 1832 master.cpp:2891] Authorizing framework principal '' to launch task 2 as user 'root' [12:21:23]W: [Step 11/11] I0324 12:21:23.902684 1829 master.hpp:177] Adding task 0 with resources cpus(*):1; mem(*):1 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 (ip-172-30-2-102.mesosphere.io) [12:21:23]W: [Step 11/11] I0324 12:21:23.902770 1829 master.cpp:3773] Launching task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 with resources cpus(*):1; mem(*):1 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:23]W: [Step 11/11] I0324 12:21:23.903033 1830 slave.cpp:1496] Got assigned task 0 for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.903141 1830 slave.cpp:5422] Checkpointing FrameworkInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/framework.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.903297 1829 master.hpp:177] Adding task 1 with resources cpus(*):1; mem(*):1 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 (ip-172-30-2-102.mesosphere.io) [12:21:23]W: [Step 11/11] I0324 12:21:23.903337 1830 slave.cpp:5433] Checkpointing framework pid 'scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940' to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/framework.pid' [12:21:23]W: [Step 11/11] I0324 12:21:23.903374 1829 master.cpp:3773] Launching task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 with resources cpus(*):1; mem(*):1 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:23]W: [Step 11/11] I0324 12:21:23.903493 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.903841 1829 master.hpp:177] Adding task 2 with resources cpus(*):1; mem(*):1 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 (ip-172-30-2-102.mesosphere.io) [12:21:23]W: [Step 11/11] I0324 12:21:23.903903 1830 slave.cpp:1496] Got assigned task 1 for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.903918 1829 master.cpp:3773] Launching task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 with resources cpus(*):1; mem(*):1 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:23]W: [Step 11/11] I0324 12:21:23.903966 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.904336 1830 slave.cpp:1615] Launching task 0 for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.904394 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.904453 1833 hierarchical.cpp:890] Recovered cpus(*):997; mem(*):997; disk(*):15342; ports(*):[31000-32000] (total: cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000], allocated: cpus(*):3; mem(*):3) on slave b8970495-be32-4130-94ae-464e22ce731b-S0 from framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.904496 1833 hierarchical.cpp:927] Framework b8970495-be32-4130-94ae-464e22ce731b-0000 filtered slave b8970495-be32-4130-94ae-464e22ce731b-S0 for 5secs [12:21:23]W: [Step 11/11] I0324 12:21:23.904847 1830 paths.cpp:528] Trying to chown '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38' to user 'root' [12:21:23]W: [Step 11/11] I0324 12:21:23.908491 1830 slave.cpp:5874] Checkpointing ExecutorInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/executor.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.908860 1830 slave.cpp:5502] Launching executor 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38' [12:21:23]W: [Step 11/11] I0324 12:21:23.909257 1829 containerizer.cpp:666] Starting container '48acf03a-dcd8-467f-85b4-fc8fee40ba38' for executor '0' of framework 'b8970495-be32-4130-94ae-464e22ce731b-0000' [12:21:23]W: [Step 11/11] I0324 12:21:23.909309 1830 slave.cpp:5897] Checkpointing TaskInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38/tasks/0/task.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.909528 1830 slave.cpp:1833] Queuing task '0' for executor '0' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.909682 1830 slave.cpp:1496] Got assigned task 2 for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.909746 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.910136 1830 slave.cpp:1615] Launching task 1 for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.910197 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.910572 1830 paths.cpp:528] Trying to chown '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e' to user 'root' [12:21:23]W: [Step 11/11] I0324 12:21:23.913755 1830 slave.cpp:5874] Checkpointing ExecutorInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/executor.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.914136 1830 slave.cpp:5502] Launching executor 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e' [12:21:23]W: [Step 11/11] I0324 12:21:23.914520 1830 slave.cpp:5897] Checkpointing TaskInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e/tasks/1/task.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.914659 1829 linux_launcher.cpp:304] Cloning child process with flags = [12:21:23]W: [Step 11/11] I0324 12:21:23.914712 1830 slave.cpp:1833] Queuing task '1' for executor '1' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.914791 1830 slave.cpp:880] Successfully attached file '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38' [12:21:23]W: [Step 11/11] I0324 12:21:23.914821 1830 slave.cpp:1615] Launching task 2 for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.914876 1830 resources.cpp:572] Parsing resources as JSON failed: cpus:0.1;mem:32 [12:21:23]W: [Step 11/11] Trying semicolon-delimited string format instead [12:21:23]W: [Step 11/11] I0324 12:21:23.915982 1830 paths.cpp:528] Trying to chown '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5' to user 'root' [12:21:23]W: [Step 11/11] I0324 12:21:23.917361 1829 containerizer.cpp:1147] Checkpointing executor's forked pid 5417 to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38/pids/forked.pid' [12:21:23]W: [Step 11/11] I0324 12:21:23.918423 1829 containerizer.cpp:666] Starting container '91f578e7-2c54-407b-9c6e-9accef600d4e' for executor '1' of framework 'b8970495-be32-4130-94ae-464e22ce731b-0000' [12:21:23]W: [Step 11/11] I0324 12:21:23.919504 1830 slave.cpp:5874] Checkpointing ExecutorInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/executor.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.919936 1830 slave.cpp:5502] Launching executor 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5' [12:21:23]W: [Step 11/11] I0324 12:21:23.920342 1830 slave.cpp:5897] Checkpointing TaskInfo to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5/tasks/2/task.info' [12:21:23]W: [Step 11/11] I0324 12:21:23.920537 1830 slave.cpp:1833] Queuing task '2' for executor '2' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:23]W: [Step 11/11] I0324 12:21:23.920647 1830 slave.cpp:880] Successfully attached file '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e' [12:21:23]W: [Step 11/11] I0324 12:21:23.920680 1830 slave.cpp:880] Successfully attached file '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5' [12:21:23]W: [Step 11/11] I0324 12:21:23.921154 1833 fetcher.cpp:323] Starting to fetch URIs for container: 48acf03a-dcd8-467f-85b4-fc8fee40ba38, directory: /tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38 [12:21:23]W: [Step 11/11] I0324 12:21:23.921887 1829 containerizer.cpp:666] Starting container '9fca1443-da37-4551-b061-5f619f9803a5' for executor '2' of framework 'b8970495-be32-4130-94ae-464e22ce731b-0000' [12:21:23]W: [Step 11/11] I0324 12:21:23.924443 1833 fetcher.cpp:912] Created cache entry 'root@http://172.30.2.102:40940/(2699)/mesos-fetcher-test-archive.tgz' with file: c1-mesos-fetc_rchive.tgz [12:21:23]W: [Step 11/11] I0324 12:21:23.925190 1828 fetcher.cpp:271] Fetching size for URI: http://172.30.2.102:40940/(2699)/mesos-fetcher-test-archive.tgz [12:21:23]W: [Step 11/11] I0324 12:21:23.926982 1829 linux_launcher.cpp:304] Cloning child process with flags = [12:21:23]W: [Step 11/11] I0324 12:21:23.928725 1829 containerizer.cpp:1147] Checkpointing executor's forked pid 5433 to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e/pids/forked.pid' [12:21:23]W: [Step 11/11] I0324 12:21:23.931329 1829 linux_launcher.cpp:304] Cloning child process with flags = [12:21:23]W: [Step 11/11] I0324 12:21:23.932957 1829 containerizer.cpp:1147] Checkpointing executor's forked pid 5437 to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5/pids/forked.pid' [12:21:23]W: [Step 11/11] I0324 12:21:23.934474 1832 fetcher.cpp:323] Starting to fetch URIs for container: 91f578e7-2c54-407b-9c6e-9accef600d4e, directory: /tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e [12:21:23]W: [Step 11/11] I0324 12:21:23.940582 1832 fetcher.cpp:323] Starting to fetch URIs for container: 9fca1443-da37-4551-b061-5f619f9803a5, directory: /tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5 [12:21:23]W: [Step 11/11] I0324 12:21:23.944748 1834 process.cpp:3136] Handling HTTP event for process '(2699)' with path: '/(2699)/mesos-fetcher-test-archive.tgz' [12:21:23]W: [Step 11/11] I0324 12:21:23.945168 1829 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/mesos-fetcher-test-assets/mesos-fetcher-test-archive.tgz' with length 10240 [12:21:23] : [Step 11/11] HTTP/1.1 200 OK [12:21:23] : [Step 11/11] Date: Thu, 24 Mar 2016 12:21:23 GMT [12:21:23] : [Step 11/11] Content-Length: 10240 [12:21:23] : [Step 11/11] [12:21:23]W: [Step 11/11] I0324 12:21:23.950675 1832 fetcher.cpp:785] Fetching URIs using command '/mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-fetcher' [12:21:23]W: [Step 11/11] I0324 12:21:23.952643 1832 fetcher.cpp:687] Claiming fetcher cache space for: root@http://172.30.2.102:40940/(2699)/mesos-fetcher-test-archive.tgz [12:21:23]W: [Step 11/11] I0324 12:21:23.952677 1832 fetcher.cpp:1132] Claimed cache space: 10KB, now using: 10KB [12:21:23]W: [Step 11/11] I0324 12:21:23.958264 1834 fetcher.cpp:785] Fetching URIs using command '/mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-fetcher' [12:21:24]W: [Step 11/11] I0324 12:21:24.008957 1831 process.cpp:3136] Handling HTTP event for process '(2699)' with path: '/(2699)/mesos-fetcher-test-archive.tgz' [12:21:24]W: [Step 11/11] I0324 12:21:24.009239 1830 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/mesos-fetcher-test-assets/mesos-fetcher-test-archive.tgz' with length 10240 [12:21:24]W: [Step 11/11] I0324 12:21:24.009517 1829 process.cpp:3136] Handling HTTP event for process '(2699)' with path: '/(2699)/mesos-fetcher-test-archive.tgz' [12:21:24]W: [Step 11/11] I0324 12:21:24.009706 1830 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/mesos-fetcher-test-assets/mesos-fetcher-test-archive.tgz' with length 10240 [12:21:24]W: [Step 11/11] I0324 12:21:24.013797 1828 process.cpp:3136] Handling HTTP event for process '(2699)' with path: '/(2699)/mesos-fetcher-test-cmd' [12:21:24]W: [Step 11/11] I0324 12:21:24.014083 1833 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/mesos-fetcher-test-assets/mesos-fetcher-test-cmd' with length 30 [12:21:24]W: [Step 11/11] I0324 12:21:24.014401 1834 process.cpp:3136] Handling HTTP event for process '(2699)' with path: '/(2699)/mesos-fetcher-test-cmd' [12:21:24]W: [Step 11/11] I0324 12:21:24.014585 1830 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/mesos-fetcher-test-assets/mesos-fetcher-test-cmd' with length 30 [12:21:24]W: [Step 11/11] I0324 12:21:24.115496 1833 fetcher.cpp:1144] Released cache space: 0B, now using: 10KB [12:21:24]W: [Step 11/11] I0324 12:21:24.120621 1830 fetcher.cpp:785] Fetching URIs using command '/mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-fetcher' [12:21:24]W: [Step 11/11] WARNING: Logging before InitGoogleLogging() is written to STDERR [12:21:24]W: [Step 11/11] I0324 12:21:24.165174 5417 process.cpp:986] libprocess is initialized on 172.30.2.102:33037 for 8 cpus [12:21:24]W: [Step 11/11] WARNING: Logging before InitGoogleLogging() is written to STDERR [12:21:24]W: [Step 11/11] I0324 12:21:24.165351 5433 process.cpp:986] libprocess is initialized on 172.30.2.102:37320 for 8 cpus [12:21:24]W: [Step 11/11] I0324 12:21:24.166008 5417 logging.cpp:195] Logging to STDERR [12:21:24]W: [Step 11/11] I0324 12:21:24.166148 5433 logging.cpp:195] Logging to STDERR [12:21:24]W: [Step 11/11] I0324 12:21:24.167593 5417 exec.cpp:150] Version: 0.29.0 [12:21:24]W: [Step 11/11] I0324 12:21:24.167649 5433 exec.cpp:150] Version: 0.29.0 [12:21:24]W: [Step 11/11] I0324 12:21:24.173698 5550 exec.cpp:200] Executor started at: executor(1)@172.30.2.102:37320 with pid 5433 [12:21:24]W: [Step 11/11] I0324 12:21:24.175462 1831 slave.cpp:2778] Got registration for executor '1' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.176005 1831 slave.cpp:2864] Checkpointing executor pid 'executor(1)@172.30.2.102:37320' to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e/pids/libprocess.pid' [12:21:24]W: [Step 11/11] I0324 12:21:24.177527 1829 slave.cpp:1998] Sending queued task '1' to executor '1' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.177692 5543 exec.cpp:200] Executor started at: executor(1)@172.30.2.102:33037 with pid 5417 [12:21:24]W: [Step 11/11] I0324 12:21:24.177757 5546 exec.cpp:225] Executor registered on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.177930 1831 process.cpp:3136] Handling HTTP event for process '(2699)' with path: '/(2699)/mesos-fetcher-test-cmd' [12:21:24]W: [Step 11/11] I0324 12:21:24.178161 1830 process.cpp:1176] Sending file at '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/mesos-fetcher-test-assets/mesos-fetcher-test-cmd' with length 30 [12:21:24]W: [Step 11/11] I0324 12:21:24.179381 1831 slave.cpp:2778] Got registration for executor '0' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.179599 5546 exec.cpp:237] Executor::registered took 375204ns [12:21:24]W: [Step 11/11] I0324 12:21:24.179776 1831 slave.cpp:2864] Checkpointing executor pid 'executor(1)@172.30.2.102:33037' to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38/pids/libprocess.pid' [12:21:24]W: [Step 11/11] I0324 12:21:24.179929 5546 exec.cpp:312] Executor asked to run task '1' [12:21:24]W: [Step 11/11] I0324 12:21:24.180100 5546 exec.cpp:321] Executor::launchTask took 144507ns [12:21:24] : [Step 11/11] Registered executor on ip-172-30-2-102.mesosphere.io [12:21:24] : [Step 11/11] Starting task 1 [12:21:24]W: [Step 11/11] I0324 12:21:24.181061 5553 exec.cpp:225] Executor registered on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24] : [Step 11/11] Forked command at 5558 [12:21:24]W: [Step 11/11] I0324 12:21:24.181191 1832 slave.cpp:1998] Sending queued task '0' to executor '0' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:33037 [12:21:24] : [Step 11/11] sh -c './mesos-fetcher-test-acmd 1' [12:21:24]W: [Step 11/11] I0324 12:21:24.182634 5553 exec.cpp:237] Executor::registered took 202046ns [12:21:24]W: [Step 11/11] I0324 12:21:24.182834 5553 exec.cpp:312] Executor asked to run task '0' [12:21:24]W: [Step 11/11] I0324 12:21:24.182909 5553 exec.cpp:321] Executor::launchTask took 55128ns [12:21:24]W: [Step 11/11] I0324 12:21:24.182833 5542 exec.cpp:535] Executor sending status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24] : [Step 11/11] Registered executor on ip-172-30-2-102.mesosphere.io [12:21:24] : [Step 11/11] Starting task 0 [12:21:24] : [Step 11/11] sh -c './mesos-fetcher-test-cmd 0' [12:21:24] : [Step 11/11] Forked command at 5560 [12:21:24]W: [Step 11/11] I0324 12:21:24.183768 1833 slave.cpp:3137] Handling status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.184415 5553 exec.cpp:535] Executor sending status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.185176 1829 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.185221 1829 status_update_manager.cpp:497] Creating StatusUpdate stream for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.185384 1832 slave.cpp:3137] Handling status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.185550 1829 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.189551 1829 status_update_manager.cpp:374] Forwarding update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to the slave [12:21:24]W: [Step 11/11] I0324 12:21:24.189782 1834 slave.cpp:3535] Forwarding the update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.189852 1829 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.189888 1829 status_update_manager.cpp:497] Creating StatusUpdate stream for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.189930 1834 slave.cpp:3429] Status update manager successfully handled status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.189972 1834 slave.cpp:3445] Sending acknowledgement for status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.190119 1830 master.cpp:4947] Status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.190165 1830 master.cpp:4995] Forwarding status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.190249 1829 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.190368 1830 master.cpp:6608] Updating the state of task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) [12:21:24]W: [Step 11/11] I0324 12:21:24.190582 5546 exec.cpp:358] Executor received status update acknowledgement 3ef63353-6800-4480-87e2-65444cbdba1d for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.190664 1828 sched.cpp:982] Scheduler::statusUpdate took 180895ns [12:21:24]W: [Step 11/11] I0324 12:21:24.190906 1835 master.cpp:4102] Processing ACKNOWLEDGE call 3ef63353-6800-4480-87e2-65444cbdba1d for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.193421 1829 status_update_manager.cpp:374] Forwarding update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to the slave [12:21:24]W: [Step 11/11] I0324 12:21:24.193645 1832 slave.cpp:3535] Forwarding the update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.193720 1829 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.193783 1832 slave.cpp:3429] Status update manager successfully handled status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.193816 1832 slave.cpp:3445] Sending acknowledgement for status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.193841 1829 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.193900 1835 master.cpp:4947] Status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.193943 1835 master.cpp:4995] Forwarding status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.194156 1835 master.cpp:6608] Updating the state of task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) [12:21:24]W: [Step 11/11] I0324 12:21:24.194330 5541 exec.cpp:358] Executor received status update acknowledgement 3e370ee0-17d7-43cd-bb7a-337c3112b760 for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.194347 1831 sched.cpp:982] Scheduler::statusUpdate took 117620ns [12:21:24]W: [Step 11/11] I0324 12:21:24.194532 1833 master.cpp:4102] Processing ACKNOWLEDGE call 3e370ee0-17d7-43cd-bb7a-337c3112b760 for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.196328 1833 slave.cpp:2547] Status update manager successfully handled status update acknowledgement (UUID: 3ef63353-6800-4480-87e2-65444cbdba1d) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.196349 1829 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.196454 1829 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.198565 1834 slave.cpp:2547] Status update manager successfully handled status update acknowledgement (UUID: 3e370ee0-17d7-43cd-bb7a-337c3112b760) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] WARNING: Logging before InitGoogleLogging() is written to STDERR [12:21:24]W: [Step 11/11] I0324 12:21:24.263320 5437 process.cpp:986] libprocess is initialized on 172.30.2.102:48576 for 8 cpus [12:21:24]W: [Step 11/11] I0324 12:21:24.263938 5437 logging.cpp:195] Logging to STDERR [12:21:24]W: [Step 11/11] I0324 12:21:24.265460 5437 exec.cpp:150] Version: 0.29.0 [12:21:24]W: [Step 11/11] I0324 12:21:24.271081 5582 exec.cpp:200] Executor started at: executor(1)@172.30.2.102:48576 with pid 5437 [12:21:24]W: [Step 11/11] I0324 12:21:24.272682 1828 slave.cpp:2778] Got registration for executor '2' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.273097 1828 slave.cpp:2864] Checkpointing executor pid 'executor(1)@172.30.2.102:48576' to '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5/pids/libprocess.pid' [12:21:24]W: [Step 11/11] I0324 12:21:24.274411 5581 exec.cpp:225] Executor registered on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.274515 1830 slave.cpp:1998] Sending queued task '2' to executor '2' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.275733 5581 exec.cpp:237] Executor::registered took 235607ns [12:21:24]W: [Step 11/11] I0324 12:21:24.275971 5581 exec.cpp:312] Executor asked to run task '2' [12:21:24]W: [Step 11/11] I0324 12:21:24.276083 5581 exec.cpp:321] Executor::launchTask took 90509ns [12:21:24] : [Step 11/11] Registered executor on ip-172-30-2-102.mesosphere.io [12:21:24] : [Step 11/11] Starting task 2 [12:21:24] : [Step 11/11] Forked command at 5584 [12:21:24] : [Step 11/11] sh -c './mesos-fetcher-test-acmd 2' [12:21:24]W: [Step 11/11] I0324 12:21:24.278383 5576 exec.cpp:535] Executor sending status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.279162 1835 slave.cpp:3137] Handling status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.280331 1828 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.280388 1828 status_update_manager.cpp:497] Creating StatusUpdate stream for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.280694 1828 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24] : [Step 11/11] Command exited with status 0 (pid: 5558) [12:21:24]W: [Step 11/11] I0324 12:21:24.283119 1828 status_update_manager.cpp:374] Forwarding update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to the slave [12:21:24]W: [Step 11/11] I0324 12:21:24.283371 1829 slave.cpp:3535] Forwarding the update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.283545 1829 slave.cpp:3429] Status update manager successfully handled status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.283579 1829 slave.cpp:3445] Sending acknowledgement for status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.283710 1833 master.cpp:4947] Status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.283759 1833 master.cpp:4995] Forwarding status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.283947 1833 master.cpp:6608] Updating the state of task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING) [12:21:24]W: [Step 11/11] I0324 12:21:24.284056 5580 exec.cpp:358] Executor received status update acknowledgement a54c14b2-8b2a-4ed9-b351-8e646392c708 for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.284260 1830 sched.cpp:982] Scheduler::statusUpdate took 169140ns [12:21:24]W: [Step 11/11] I0324 12:21:24.284484 1832 master.cpp:4102] Processing ACKNOWLEDGE call a54c14b2-8b2a-4ed9-b351-8e646392c708 for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.284610 5550 exec.cpp:535] Executor sending status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.284749 1832 status_update_manager.cpp:392] Received status update acknowledgement (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.284870 1832 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_RUNNING (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.285192 1834 slave.cpp:3137] Handling status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.286197 1835 slave.cpp:5812] Terminating task 1 [12:21:24] : [Step 11/11] Command exited with status 0 (pid: 5560) [12:21:24]W: [Step 11/11] I0324 12:21:24.286974 1829 slave.cpp:2547] Status update manager successfully handled status update acknowledgement (UUID: a54c14b2-8b2a-4ed9-b351-8e646392c708) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.287277 1830 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.287331 1830 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.287458 5545 exec.cpp:535] Executor sending status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.288038 1835 slave.cpp:3137] Handling status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.288959 1828 slave.cpp:5812] Terminating task 0 [12:21:24]W: [Step 11/11] I0324 12:21:24.289489 1830 status_update_manager.cpp:374] Forwarding update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to the slave [12:21:24]W: [Step 11/11] I0324 12:21:24.289927 1835 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.289969 1835 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.290026 1829 slave.cpp:3535] Forwarding the update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.290190 1829 slave.cpp:3429] Status update manager successfully handled status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.290225 1829 slave.cpp:3445] Sending acknowledgement for status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.290323 1831 master.cpp:4947] Status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.290356 1831 master.cpp:4995] Forwarding status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.290487 1831 master.cpp:6608] Updating the state of task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (latest state: TASK_FINISHED, status update state: TASK_FINISHED) [12:21:24]W: [Step 11/11] I0324 12:21:24.290741 1828 hierarchical.cpp:890] Recovered cpus(*):1; mem(*):1 (total: cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):2) on slave b8970495-be32-4130-94ae-464e22ce731b-S0 from framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.290812 1832 sched.cpp:982] Scheduler::statusUpdate took 217034ns [12:21:24]W: [Step 11/11] I0324 12:21:24.290815 5546 exec.cpp:358] Executor received status update acknowledgement 194c02b6-9857-4864-9781-70442b343c0e for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.291024 1831 master.cpp:4102] Processing ACKNOWLEDGE call 194c02b6-9857-4864-9781-70442b343c0e for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.291086 1831 master.cpp:6674] Removing task 1 with resources cpus(*):1; mem(*):1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.294679 1835 status_update_manager.cpp:374] Forwarding update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to the slave [12:21:24]W: [Step 11/11] I0324 12:21:24.294895 1831 slave.cpp:3535] Forwarding the update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.294971 1835 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.295078 1831 slave.cpp:3429] Status update manager successfully handled status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.295111 1831 slave.cpp:3445] Sending acknowledgement for status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.295111 1835 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FINISHED (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.295192 1829 master.cpp:4947] Status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.295235 1829 master.cpp:4995] Forwarding status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.295413 1829 master.cpp:6608] Updating the state of task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (latest state: TASK_FINISHED, status update state: TASK_FINISHED) [12:21:24]W: [Step 11/11] I0324 12:21:24.295601 5551 exec.cpp:358] Executor received status update acknowledgement 0a0dcb0c-9767-4671-848d-38df4599dc8c for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.295723 1834 sched.cpp:982] Scheduler::statusUpdate took 206183ns [12:21:24]W: [Step 11/11] I0324 12:21:24.295760 1833 hierarchical.cpp:890] Recovered cpus(*):1; mem(*):1 (total: cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000], allocated: cpus(*):1; mem(*):1) on slave b8970495-be32-4130-94ae-464e22ce731b-S0 from framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.295907 1833 master.cpp:4102] Processing ACKNOWLEDGE call 0a0dcb0c-9767-4671-848d-38df4599dc8c for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.295950 1833 master.cpp:6674] Removing task 0 with resources cpus(*):1; mem(*):1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.297143 1835 status_update_manager.cpp:528] Cleaning up status update stream for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.297370 1828 slave.cpp:2547] Status update manager successfully handled status update acknowledgement (UUID: 194c02b6-9857-4864-9781-70442b343c0e) for task 1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.297394 1835 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.297407 1828 slave.cpp:5853] Completing task 1 [12:21:24]W: [Step 11/11] I0324 12:21:24.297498 1835 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FINISHED (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.299329 1835 status_update_manager.cpp:528] Cleaning up status update stream for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.299540 1832 slave.cpp:2547] Status update manager successfully handled status update acknowledgement (UUID: 0a0dcb0c-9767-4671-848d-38df4599dc8c) for task 0 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.299581 1832 slave.cpp:5853] Completing task 0 [12:21:24] : [Step 11/11] Command exited with status 0 (pid: 5584) [12:21:24]W: [Step 11/11] I0324 12:21:24.378903 5575 exec.cpp:535] Executor sending status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.379477 1831 slave.cpp:3137] Handling status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.380447 1833 slave.cpp:5812] Terminating task 2 [12:21:24]W: [Step 11/11] I0324 12:21:24.381443 1829 status_update_manager.cpp:320] Received status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.381489 1829 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.383627 1829 status_update_manager.cpp:374] Forwarding update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to the slave [12:21:24]W: [Step 11/11] I0324 12:21:24.383888 1831 slave.cpp:3535] Forwarding the update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.384073 1831 slave.cpp:3429] Status update manager successfully handled status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.384110 1831 slave.cpp:3445] Sending acknowledgement for status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 to executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.384201 1828 master.cpp:4947] Status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 from slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.384246 1828 master.cpp:4995] Forwarding status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.384428 1828 master.cpp:6608] Updating the state of task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (latest state: TASK_FINISHED, status update state: TASK_FINISHED) [12:21:24]W: [Step 11/11] I0324 12:21:24.384594 5582 exec.cpp:358] Executor received status update acknowledgement ccf8a66e-f2be-4cb5-a273-40f0533bb08d for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.384651 1835 sched.cpp:982] Scheduler::statusUpdate took 154943ns [12:21:24]W: [Step 11/11] I0324 12:21:24.384781 1833 hierarchical.cpp:890] Recovered cpus(*):1; mem(*):1 (total: cpus(*):1000; mem(*):1000; disk(*):15342; ports(*):[31000-32000], allocated: ) on slave b8970495-be32-4130-94ae-464e22ce731b-S0 from framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.384943 1832 master.cpp:4102] Processing ACKNOWLEDGE call ccf8a66e-f2be-4cb5-a273-40f0533bb08d for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.385004 1832 master.cpp:6674] Removing task 2 with resources cpus(*):1; mem(*):1 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 on slave b8970495-be32-4130-94ae-464e22ce731b-S0 at slave(57)@172.30.2.102:40940 (ip-172-30-2-102.mesosphere.io) [12:21:24]W: [Step 11/11] I0324 12:21:24.385331 1832 status_update_manager.cpp:392] Received status update acknowledgement (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.385463 1832 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_FINISHED (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.385632 1814 sched.cpp:1904] Asked to stop the driver [12:21:24]W: [Step 11/11] I0324 12:21:24.385711 1834 sched.cpp:1144] Stopping framework 'b8970495-be32-4130-94ae-464e22ce731b-0000' [12:21:24]W: [Step 11/11] I0324 12:21:24.385888 1828 master.cpp:6081] Processing TEARDOWN call for framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.385921 1828 master.cpp:6093] Removing framework b8970495-be32-4130-94ae-464e22ce731b-0000 (default) at scheduler-69a60d31-9be6-4f35-85ab-2116f0b92279@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.386042 1833 hierarchical.cpp:375] Deactivated framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.386114 1833 slave.cpp:2214] Asked to shut down framework b8970495-be32-4130-94ae-464e22ce731b-0000 by master@172.30.2.102:40940 [12:21:24]W: [Step 11/11] I0324 12:21:24.386147 1833 slave.cpp:2239] Shutting down framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.386198 1833 slave.cpp:4333] Shutting down executor '0' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.386389 1833 slave.cpp:4333] Shutting down executor '1' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.386409 1828 master.cpp:1083] Master terminating [12:21:24]W: [Step 11/11] I0324 12:21:24.386468 1834 hierarchical.cpp:326] Removed framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.386585 1833 slave.cpp:4333] Shutting down executor '2' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.386833 1834 hierarchical.cpp:505] Removed slave b8970495-be32-4130-94ae-464e22ce731b-S0 [12:21:24]W: [Step 11/11] I0324 12:21:24.386836 5549 exec.cpp:399] Executor asked to shutdown [12:21:24]W: [Step 11/11] I0324 12:21:24.386972 5549 exec.cpp:414] Executor::shutdown took 17271ns [12:21:24]W: [Step 11/11] I0324 12:21:24.386982 5551 exec.cpp:91] Scheduling shutdown of the executor in 5secs [12:21:24]W: [Step 11/11] I0324 12:21:24.387181 1829 slave.cpp:3663] master@172.30.2.102:40940 exited [12:21:24]W: [Step 11/11] W0324 12:21:24.387202 1829 slave.cpp:3666] Master disconnected! Waiting for a new master to be elected [12:21:24]W: [Step 11/11] I0324 12:21:24.387269 5539 exec.cpp:399] Executor asked to shutdown [12:21:24]W: [Step 11/11] I0324 12:21:24.387326 5578 exec.cpp:399] Executor asked to shutdown [12:21:24]W: [Step 11/11] I0324 12:21:24.387413 5539 exec.cpp:414] Executor::shutdown took 18284ns [12:21:24]W: [Step 11/11] I0324 12:21:24.387437 5578 exec.cpp:414] Executor::shutdown took 13644ns [12:21:24]W: [Step 11/11] I0324 12:21:24.387446 5539 exec.cpp:91] Scheduling shutdown of the executor in 5secs [12:21:24]W: [Step 11/11] I0324 12:21:24.387547 5578 exec.cpp:91] Scheduling shutdown of the executor in 5secs [12:21:24]W: [Step 11/11] I0324 12:21:24.388180 1832 status_update_manager.cpp:528] Cleaning up status update stream for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.388475 1833 slave.cpp:2547] Status update manager successfully handled status update acknowledgement (UUID: ccf8a66e-f2be-4cb5-a273-40f0533bb08d) for task 2 of framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.388510 1833 slave.cpp:5853] Completing task 2 [12:21:24]W: [Step 11/11] I0324 12:21:24.391641 1829 containerizer.cpp:1421] Destroying container '48acf03a-dcd8-467f-85b4-fc8fee40ba38' [12:21:24]W: [Step 11/11] I0324 12:21:24.393930 1828 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/48acf03a-dcd8-467f-85b4-fc8fee40ba38 [12:21:24]W: [Step 11/11] I0324 12:21:24.395978 1832 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/48acf03a-dcd8-467f-85b4-fc8fee40ba38 after 2.001152ms [12:21:24]W: [Step 11/11] I0324 12:21:24.398180 1835 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/48acf03a-dcd8-467f-85b4-fc8fee40ba38 [12:21:24]W: [Step 11/11] I0324 12:21:24.400089 1835 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/48acf03a-dcd8-467f-85b4-fc8fee40ba38 after 1.869056ms [12:21:24]W: [Step 11/11] I0324 12:21:24.401280 1830 slave.cpp:3663] executor(1)@172.30.2.102:33037 exited [12:21:24]W: [Step 11/11] I0324 12:21:24.417935 1830 containerizer.cpp:1656] Executor for container '48acf03a-dcd8-467f-85b4-fc8fee40ba38' has exited [12:21:24]W: [Step 11/11] I0324 12:21:24.420359 1829 provisioner.cpp:306] Ignoring destroy request for unknown container 48acf03a-dcd8-467f-85b4-fc8fee40ba38 [12:21:24]W: [Step 11/11] I0324 12:21:24.420590 1829 slave.cpp:4021] Executor '0' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 terminated with signal Killed [12:21:24]W: [Step 11/11] I0324 12:21:24.420627 1829 slave.cpp:4125] Cleaning up executor '0' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:33037 [12:21:24]W: [Step 11/11] I0324 12:21:24.420917 1831 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38' for gc 6.99999512924741days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.421068 1831 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0' for gc 6.99999512795259days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.421098 1835 containerizer.cpp:1421] Destroying container '91f578e7-2c54-407b-9c6e-9accef600d4e' [12:21:24]W: [Step 11/11] I0324 12:21:24.421167 1831 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0/runs/48acf03a-dcd8-467f-85b4-fc8fee40ba38' for gc 6.99999512708444days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.421272 1831 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/0' for gc 6.9999951262963days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.423120 1831 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/91f578e7-2c54-407b-9c6e-9accef600d4e [12:21:24]W: [Step 11/11] I0324 12:21:24.424975 1832 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/91f578e7-2c54-407b-9c6e-9accef600d4e after 1.809152ms [12:21:24]W: [Step 11/11] I0324 12:21:24.427112 1830 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/91f578e7-2c54-407b-9c6e-9accef600d4e [12:21:24]W: [Step 11/11] I0324 12:21:24.428988 1830 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/91f578e7-2c54-407b-9c6e-9accef600d4e after 1.835264ms [12:21:24]W: [Step 11/11] I0324 12:21:24.429930 1829 slave.cpp:3663] executor(1)@172.30.2.102:37320 exited [12:21:24]W: [Step 11/11] I0324 12:21:24.518996 1828 containerizer.cpp:1656] Executor for container '91f578e7-2c54-407b-9c6e-9accef600d4e' has exited [12:21:24]W: [Step 11/11] I0324 12:21:24.521729 1832 provisioner.cpp:306] Ignoring destroy request for unknown container 91f578e7-2c54-407b-9c6e-9accef600d4e [12:21:24]W: [Step 11/11] I0324 12:21:24.521989 1834 slave.cpp:4021] Executor '1' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 terminated with signal Killed [12:21:24]W: [Step 11/11] I0324 12:21:24.522037 1834 slave.cpp:4125] Cleaning up executor '1' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:37320 [12:21:24]W: [Step 11/11] I0324 12:21:24.522341 1828 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e' for gc 6.99999395535704days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.522506 1828 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1' for gc 6.99999395392296days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.522557 1835 containerizer.cpp:1421] Destroying container '9fca1443-da37-4551-b061-5f619f9803a5' [12:21:24]W: [Step 11/11] I0324 12:21:24.522663 1828 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1/runs/91f578e7-2c54-407b-9c6e-9accef600d4e' for gc 6.99999395318222days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.522755 1828 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/1' for gc 6.99999395237333days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.524680 1834 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/9fca1443-da37-4551-b061-5f619f9803a5 [12:21:24]W: [Step 11/11] I0324 12:21:24.526599 1830 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/9fca1443-da37-4551-b061-5f619f9803a5 after 1.875968ms [12:21:24]W: [Step 11/11] I0324 12:21:24.529247 1834 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/9fca1443-da37-4551-b061-5f619f9803a5 [12:21:24]W: [Step 11/11] I0324 12:21:24.531468 1834 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/9fca1443-da37-4551-b061-5f619f9803a5 after 2.179072ms [12:21:24]W: [Step 11/11] I0324 12:21:24.533591 1828 slave.cpp:3663] executor(1)@172.30.2.102:48576 exited [12:21:24]W: [Step 11/11] I0324 12:21:24.620025 1829 containerizer.cpp:1656] Executor for container '9fca1443-da37-4551-b061-5f619f9803a5' has exited [12:21:24]W: [Step 11/11] I0324 12:21:24.622593 1834 provisioner.cpp:306] Ignoring destroy request for unknown container 9fca1443-da37-4551-b061-5f619f9803a5 [12:21:24]W: [Step 11/11] I0324 12:21:24.622856 1829 slave.cpp:4021] Executor '2' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 terminated with signal Killed [12:21:24]W: [Step 11/11] I0324 12:21:24.622902 1829 slave.cpp:4125] Cleaning up executor '2' of framework b8970495-be32-4130-94ae-464e22ce731b-0000 at executor(1)@172.30.2.102:48576 [12:21:24]W: [Step 11/11] I0324 12:21:24.623231 1835 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5' for gc 6.99999278759111days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.623402 1835 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2' for gc 6.99999278596148days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.623507 1829 slave.cpp:4213] Cleaning up framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.623500 1835 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2/runs/9fca1443-da37-4551-b061-5f619f9803a5' for gc 6.99999278500148days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.623607 1828 status_update_manager.cpp:282] Closing status update streams for framework b8970495-be32-4130-94ae-464e22ce731b-0000 [12:21:24]W: [Step 11/11] I0324 12:21:24.623620 1835 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000/executors/2' for gc 6.99999278412148days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.623718 1835 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000' for gc 6.99999278224593days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.623813 1835 gc.cpp:55] Scheduling '/tmp/FetcherCacheHttpTest_HttpMixed_oq9v2C/meta/slaves/b8970495-be32-4130-94ae-464e22ce731b-S0/frameworks/b8970495-be32-4130-94ae-464e22ce731b-0000' for gc 6.99999278146963days in the future [12:21:24]W: [Step 11/11] I0324 12:21:24.624037 1830 slave.cpp:799] Slave terminating [12:21:24] : [Step 11/11] [ OK ] FetcherCacheHttpTest.HttpMixed (807 ms)