Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
1.0.1
-
None
-
Fedora 23, SSL & libevent enabled
Description
Observed on our internal CI:
[02:56:34] : [Step 10/10] [ RUN ] ResourceOffersTest.ResourcesGetReofferedAfterTaskInfoError [02:56:49] : [Step 10/10] ../../src/tests/resource_offers_tests.cpp:277: Failure [02:56:49] : [Step 10/10] Failed to wait 15secs for offers [02:58:28]W: [Step 10/10] I0915 02:56:33.946434 29360 slave.cpp:2496] Asked to shut down framework 9ce9a215-ee08-4592-a75e-81d69e115b0b-0001 by master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.946470 29357 hierarchical.cpp:380] Deactivated framework 9ce9a215-ee08-4592-a75e-81d69e115b0b-0001 [02:58:28]W: [Step 10/10] W0915 02:56:33.946485 29360 slave.cpp:2511] Cannot shut down unknown framework 9ce9a215-ee08-4592-a75e-81d69e115b0b-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.946627 29357 hierarchical.cpp:1003] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on agent 9ce9a215-ee08-4592-a75e-81d69e115b0b-S0 from framework 9ce9a215-ee08-4592-a75e-81d69e115b0b-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.946697 29357 hierarchical.cpp:331] Removed framework 9ce9a215-ee08-4592-a75e-81d69e115b0b-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.946774 29358 slave.cpp:783] Agent terminating [02:58:28]W: [Step 10/10] I0915 02:56:33.946842 29358 master.cpp:1254] Agent 9ce9a215-ee08-4592-a75e-81d69e115b0b-S0 at slave(338)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) disconnected [02:58:28]W: [Step 10/10] I0915 02:56:33.946861 29358 master.cpp:2789] Disconnecting agent 9ce9a215-ee08-4592-a75e-81d69e115b0b-S0 at slave(338)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) [02:58:28]W: [Step 10/10] I0915 02:56:33.946888 29358 master.cpp:2808] Deactivating agent 9ce9a215-ee08-4592-a75e-81d69e115b0b-S0 at slave(338)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) [02:58:28]W: [Step 10/10] I0915 02:56:33.946988 29358 hierarchical.cpp:569] Agent 9ce9a215-ee08-4592-a75e-81d69e115b0b-S0 deactivated [02:58:28]W: [Step 10/10] I0915 02:56:33.953768 29362 master.cpp:1097] Master terminating [02:58:28] : [Step 10/10] ../../src/tests/resource_offers_tests.cpp:271: Failure [02:58:28]W: [Step 10/10] I0915 02:56:33.953980 29363 hierarchical.cpp:508] Removed agent 9ce9a215-ee08-4592-a75e-81d69e115b0b-S0 [02:58:28] : [Step 10/10] Actual function call count doesn't match EXPECT_CALL(sched2, resourceOffers(&driver2, _))... [02:58:28] : [Step 10/10] Expected: to be called at least once [02:58:28]W: [Step 10/10] I0915 02:56:33.966758 29343 cluster.cpp:157] Creating default 'local' authorizer [02:58:28] : [Step 10/10] Actual: never called - unsatisfied and active [02:58:28] : [Step 10/10] ../../src/tests/resource_offers_tests.cpp:268: Failure [02:58:28]W: [Step 10/10] I0915 02:56:33.969426 29343 leveldb.cpp:174] Opened db in 2.552034ms [02:58:28] : [Step 10/10] Actual function call count doesn't match EXPECT_CALL(sched2, registered(&driver2, _, _))... [02:58:28]W: [Step 10/10] I0915 02:56:33.969872 29343 leveldb.cpp:181] Compacted db in 427353ns [02:58:28] : [Step 10/10] Expected: to be called once [02:58:28] : [Step 10/10] Actual: never called - unsatisfied and active [02:58:28]W: [Step 10/10] I0915 02:56:33.969892 29343 leveldb.cpp:196] Created db iterator in 3824ns [02:58:28]W: [Step 10/10] I0915 02:56:33.969897 29343 leveldb.cpp:202] Seeked to beginning of db in 469ns [02:58:28]W: [Step 10/10] I0915 02:56:33.969903 29343 leveldb.cpp:271] Iterated through 0 keys in the db in 608ns [02:58:28]W: [Step 10/10] I0915 02:56:33.969914 29343 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned [02:58:28]W: [Step 10/10] I0915 02:56:33.970141 29364 recover.cpp:451] Starting replica recovery [02:58:28]W: [Step 10/10] I0915 02:56:33.970238 29360 recover.cpp:477] Replica is in EMPTY status [02:58:28]W: [Step 10/10] I0915 02:56:33.970479 29364 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(4300)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.970594 29361 recover.cpp:197] Received a recover response from a replica in EMPTY status [02:58:28]W: [Step 10/10] I0915 02:56:33.970743 29362 recover.cpp:568] Updating replica status to STARTING [02:58:28]W: [Step 10/10] I0915 02:56:33.971016 29364 master.cpp:380] Master 43b90154-1567-471a-b415-98140670e38d (ip-172-30-2-175.ec2.internal.mesosphere.io) started on 172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.971032 29364 master.cpp:382] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/vqh7ld/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/vqh7ld/master" --zk_session_timeout="10secs" [02:58:28]W: [Step 10/10] I0915 02:56:33.971199 29364 master.cpp:432] Master only allowing authenticated frameworks to register [02:58:28]W: [Step 10/10] I0915 02:56:33.971212 29364 master.cpp:446] Master only allowing authenticated agents to register [02:58:28]W: [Step 10/10] I0915 02:56:33.971220 29364 master.cpp:459] Master only allowing authenticated HTTP frameworks to register [02:58:28]W: [Step 10/10] I0915 02:56:33.971230 29364 credentials.hpp:37] Loading credentials for authentication from '/tmp/vqh7ld/credentials' [02:58:28]W: [Step 10/10] I0915 02:56:33.971350 29364 master.cpp:504] Using default 'crammd5' authenticator [02:58:28]W: [Step 10/10] I0915 02:56:33.971392 29364 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-readonly' [02:58:28]W: [Step 10/10] I0915 02:56:33.971472 29364 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-readwrite' [02:58:28]W: [Step 10/10] I0915 02:56:33.971529 29364 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-scheduler' [02:58:28]W: [Step 10/10] I0915 02:56:33.971585 29364 master.cpp:584] Authorization enabled [02:58:28]W: [Step 10/10] I0915 02:56:33.971642 29359 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 846599ns [02:58:28]W: [Step 10/10] I0915 02:56:33.971658 29361 whitelist_watcher.cpp:77] No whitelist given [02:58:28]W: [Step 10/10] I0915 02:56:33.971660 29359 replica.cpp:320] Persisted replica status to STARTING [02:58:28]W: [Step 10/10] I0915 02:56:33.971704 29363 hierarchical.cpp:149] Initialized hierarchical allocator process [02:58:28]W: [Step 10/10] I0915 02:56:33.971779 29359 recover.cpp:477] Replica is in STARTING status [02:58:28]W: [Step 10/10] I0915 02:56:33.972255 29362 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(4301)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.972352 29359 master.cpp:1855] Elected as the leading master! [02:58:28]W: [Step 10/10] I0915 02:56:33.972370 29359 master.cpp:1556] Recovering from registrar [02:58:28]W: [Step 10/10] I0915 02:56:33.972437 29357 recover.cpp:197] Received a recover response from a replica in STARTING status [02:58:28]W: [Step 10/10] I0915 02:56:33.972528 29364 registrar.cpp:332] Recovering registrar [02:58:28]W: [Step 10/10] I0915 02:56:33.972707 29362 recover.cpp:568] Updating replica status to VOTING [02:58:28]W: [Step 10/10] I0915 02:56:33.973048 29362 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 286382ns [02:58:28]W: [Step 10/10] I0915 02:56:33.973062 29362 replica.cpp:320] Persisted replica status to VOTING [02:58:28]W: [Step 10/10] I0915 02:56:33.973099 29362 recover.cpp:582] Successfully joined the Paxos group [02:58:28]W: [Step 10/10] I0915 02:56:33.973155 29362 recover.cpp:466] Recover process terminated [02:58:28]W: [Step 10/10] I0915 02:56:33.973332 29357 log.cpp:553] Attempting to start the writer [02:58:28]W: [Step 10/10] I0915 02:56:33.973745 29359 replica.cpp:493] Replica received implicit promise request from __req_res__(4302)@172.30.2.175:36619 with proposal 1 [02:58:28]W: [Step 10/10] I0915 02:56:33.974014 29359 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 249913ns [02:58:28]W: [Step 10/10] I0915 02:56:33.974028 29359 replica.cpp:342] Persisted promised to 1 [02:58:28]W: [Step 10/10] I0915 02:56:33.974238 29364 coordinator.cpp:238] Coordinator attempting to fill missing positions [02:58:28]W: [Step 10/10] I0915 02:56:33.974625 29360 replica.cpp:388] Replica received explicit promise request from __req_res__(4303)@172.30.2.175:36619 for position 0 with proposal 2 [02:58:28]W: [Step 10/10] I0915 02:56:33.974934 29360 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 282928ns [02:58:28]W: [Step 10/10] I0915 02:56:33.974948 29360 replica.cpp:708] Persisted action NOP at position 0 [02:58:28]W: [Step 10/10] I0915 02:56:33.975358 29361 replica.cpp:537] Replica received write request for position 0 from __req_res__(4304)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.975389 29361 leveldb.cpp:436] Reading position from leveldb took 10513ns [02:58:28]W: [Step 10/10] I0915 02:56:33.975682 29361 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 279997ns [02:58:28]W: [Step 10/10] I0915 02:56:33.975695 29361 replica.cpp:708] Persisted action NOP at position 0 [02:58:28]W: [Step 10/10] I0915 02:56:33.975878 29364 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:33.976152 29364 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 259088ns [02:58:28]W: [Step 10/10] I0915 02:56:33.976166 29364 replica.cpp:708] Persisted action NOP at position 0 [02:58:28]W: [Step 10/10] I0915 02:56:33.976352 29364 log.cpp:569] Writer started with ending position 0 [02:58:28]W: [Step 10/10] I0915 02:56:33.976622 29363 leveldb.cpp:436] Reading position from leveldb took 13714ns [02:58:28]W: [Step 10/10] I0915 02:56:33.976857 29363 registrar.cpp:365] Successfully fetched the registry (0B) in 4.307968ms [02:58:28]W: [Step 10/10] I0915 02:56:33.976886 29363 registrar.cpp:464] Applied 1 operations in 2432ns; attempting to update the registry [02:58:28]W: [Step 10/10] I0915 02:56:33.977079 29359 log.cpp:577] Attempting to append 235 bytes to the log [02:58:28]W: [Step 10/10] I0915 02:56:33.977154 29358 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 [02:58:28]W: [Step 10/10] I0915 02:56:33.977411 29358 replica.cpp:537] Replica received write request for position 1 from __req_res__(4305)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.977726 29358 leveldb.cpp:341] Persisting action (254 bytes) to leveldb took 293714ns [02:58:28]W: [Step 10/10] I0915 02:56:33.977741 29358 replica.cpp:708] Persisted action APPEND at position 1 [02:58:28]W: [Step 10/10] I0915 02:56:33.977988 29363 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:33.978276 29363 leveldb.cpp:341] Persisting action (256 bytes) to leveldb took 265936ns [02:58:28]W: [Step 10/10] I0915 02:56:33.978289 29363 replica.cpp:708] Persisted action APPEND at position 1 [02:58:28]W: [Step 10/10] I0915 02:56:33.978565 29360 log.cpp:596] Attempting to truncate the log to 1 [02:58:28]W: [Step 10/10] I0915 02:56:33.978564 29357 registrar.cpp:509] Successfully updated the registry in 1.662208ms [02:58:28]W: [Step 10/10] I0915 02:56:33.978626 29358 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 [02:58:28]W: [Step 10/10] I0915 02:56:33.978631 29357 registrar.cpp:395] Successfully recovered registrar [02:58:28]W: [Step 10/10] I0915 02:56:33.978790 29358 master.cpp:1664] Recovered 0 agents from the registry (196B); allowing 10mins for agents to re-register [02:58:28]W: [Step 10/10] I0915 02:56:33.978801 29360 hierarchical.cpp:176] Skipping recovery of hierarchical allocator: nothing to recover [02:58:28]W: [Step 10/10] I0915 02:56:33.979009 29358 replica.cpp:537] Replica received write request for position 2 from __req_res__(4306)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.979312 29358 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 280542ns [02:58:28]W: [Step 10/10] I0915 02:56:33.979324 29358 replica.cpp:708] Persisted action TRUNCATE at position 2 [02:58:28]W: [Step 10/10] I0915 02:56:33.979624 29364 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:33.979923 29364 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 282451ns [02:58:28]W: [Step 10/10] I0915 02:56:33.979943 29364 leveldb.cpp:399] Deleting ~1 keys from leveldb took 8130ns [02:58:28]W: [Step 10/10] I0915 02:56:33.979950 29364 replica.cpp:708] Persisted action TRUNCATE at position 2 [02:58:28]W: [Step 10/10] I0915 02:56:33.981307 29343 containerizer.cpp:217] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni [02:58:28]W: [Step 10/10] I0915 02:56:33.985875 29343 linux_launcher.cpp:104] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher [02:58:28]W: [Step 10/10] I0915 02:56:33.986804 29343 cluster.cpp:436] Creating default 'local' authorizer [02:58:28]W: [Step 10/10] I0915 02:56:33.987203 29363 slave.cpp:208] Mesos agent started on (339)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.987215 29363 slave.cpp:209] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JfC3wq/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JfC3wq/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_credentials="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JfC3wq/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JfC3wq" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JzbAWj" --xfs_project_range="[5000-10000]" [02:58:28]W: [Step 10/10] I0915 02:56:33.987462 29363 credentials.hpp:86] Loading credential for authentication from '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JfC3wq/credential' [02:58:28]W: [Step 10/10] I0915 02:56:33.987514 29363 slave.cpp:346] Agent using credential for: test-principal [02:58:28]W: [Step 10/10] I0915 02:56:33.987524 29363 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JfC3wq/http_credentials' [02:58:28]W: [Step 10/10] I0915 02:56:33.987591 29363 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readonly' [02:58:28]W: [Step 10/10] I0915 02:56:33.987663 29363 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite' [02:58:28]W: [Step 10/10] I0915 02:56:33.987715 29343 sched.cpp:226] Version: 1.1.0 [02:58:28]W: [Step 10/10] I0915 02:56:33.987874 29360 sched.cpp:330] New master detected at master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.987906 29360 sched.cpp:396] Authenticating with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.987918 29360 sched.cpp:403] Using default CRAM-MD5 authenticatee [02:58:28]W: [Step 10/10] I0915 02:56:33.987974 29363 slave.cpp:529] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] [02:58:28]W: [Step 10/10] I0915 02:56:33.988001 29358 authenticatee.cpp:121] Creating new client SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:33.988019 29363 slave.cpp:537] Agent attributes: [ ] [02:58:28]W: [Step 10/10] I0915 02:56:33.988031 29363 slave.cpp:542] Agent hostname: ip-172-30-2-175.ec2.internal.mesosphere.io [02:58:28]W: [Step 10/10] I0915 02:56:33.988090 29358 master.cpp:6234] Authenticating scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.988217 29360 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(710)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.988312 29359 authenticator.cpp:98] Creating new server SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:33.988441 29361 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 [02:58:28]W: [Step 10/10] I0915 02:56:33.988454 29360 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JzbAWj/meta' [02:58:28]W: [Step 10/10] I0915 02:56:33.988457 29361 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:33.988507 29361 authenticator.cpp:204] Received SASL authentication start [02:58:28]W: [Step 10/10] I0915 02:56:33.988548 29361 authenticator.cpp:326] Authentication requires more steps [02:58:28]W: [Step 10/10] I0915 02:56:33.988631 29363 authenticatee.cpp:259] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:33.988648 29358 status_update_manager.cpp:203] Recovering status update manager [02:58:28]W: [Step 10/10] I0915 02:56:33.988687 29363 authenticator.cpp:232] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:33.988701 29363 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [02:58:28]W: [Step 10/10] I0915 02:56:33.988713 29363 auxprop.cpp:181] Looking up auxiliary property '*userPassword' [02:58:28]W: [Step 10/10] I0915 02:56:33.988713 29358 containerizer.cpp:578] Recovering containerizer [02:58:28]W: [Step 10/10] I0915 02:56:33.988729 29363 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:33.988742 29363 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [02:58:28]W: [Step 10/10] I0915 02:56:33.988749 29363 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:33.988754 29363 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:33.988762 29363 authenticator.cpp:318] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:33.988811 29357 authenticatee.cpp:299] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:33.988827 29364 master.cpp:6264] Successfully authenticated principal 'test-principal' at scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.988836 29363 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(710)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.988988 29357 sched.cpp:502] Successfully authenticated with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.989004 29357 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.989039 29357 sched.cpp:853] Will retry registration in 359.780969ms if necessary [02:58:28]W: [Step 10/10] I0915 02:56:33.989079 29363 master.cpp:2429] Received SUBSCRIBE call for framework 'default' at scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.989102 29363 master.cpp:1891] Authorizing framework principal 'test-principal' to receive offers for role '*' [02:58:28]W: [Step 10/10] I0915 02:56:33.989207 29363 master.cpp:2505] Subscribing framework default with checkpointing disabled and capabilities [ ] [02:58:28]W: [Step 10/10] I0915 02:56:33.989337 29360 hierarchical.cpp:269] Added framework 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] I0915 02:56:33.989362 29360 hierarchical.cpp:1675] No allocations performed [02:58:28]W: [Step 10/10] I0915 02:56:33.989370 29360 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:33.989368 29363 sched.cpp:743] Framework registered with 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] I0915 02:56:33.989382 29360 hierarchical.cpp:1271] Performed allocation for 0 agents in 27572ns [02:58:28]W: [Step 10/10] I0915 02:56:33.989403 29363 sched.cpp:757] Scheduler::registered took 8127ns [02:58:28]W: [Step 10/10] I0915 02:56:33.989775 29357 provisioner.cpp:253] Provisioner recovery complete [02:58:28]W: [Step 10/10] I0915 02:56:33.989920 29359 slave.cpp:5197] Finished recovery [02:58:28]W: [Step 10/10] I0915 02:56:33.990099 29359 slave.cpp:5369] Querying resource estimator for oversubscribable resources [02:58:28]W: [Step 10/10] I0915 02:56:33.990167 29357 slave.cpp:911] New master detected at master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990177 29362 status_update_manager.cpp:177] Pausing sending status updates [02:58:28]W: [Step 10/10] I0915 02:56:33.990185 29357 slave.cpp:970] Authenticating with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990201 29357 slave.cpp:981] Using default CRAM-MD5 authenticatee [02:58:28]W: [Step 10/10] I0915 02:56:33.990236 29357 slave.cpp:943] Detecting new master [02:58:28]W: [Step 10/10] I0915 02:56:33.990264 29359 authenticatee.cpp:121] Creating new client SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:33.990272 29357 slave.cpp:5383] Received oversubscribable resources from the resource estimator [02:58:28]W: [Step 10/10] I0915 02:56:33.990382 29363 master.cpp:6234] Authenticating slave(339)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990424 29363 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(711)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990474 29358 authenticator.cpp:98] Creating new server SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:33.990527 29358 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 [02:58:28]W: [Step 10/10] I0915 02:56:33.990543 29358 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:33.990586 29358 authenticator.cpp:204] Received SASL authentication start [02:58:28]W: [Step 10/10] I0915 02:56:33.990631 29358 authenticator.cpp:326] Authentication requires more steps [02:58:28]W: [Step 10/10] I0915 02:56:33.990665 29358 authenticatee.cpp:259] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:33.990703 29358 authenticator.cpp:232] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:33.990720 29358 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [02:58:28]W: [Step 10/10] I0915 02:56:33.990732 29358 auxprop.cpp:181] Looking up auxiliary property '*userPassword' [02:58:28]W: [Step 10/10] I0915 02:56:33.990744 29358 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:33.990752 29358 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [02:58:28]W: [Step 10/10] I0915 02:56:33.990757 29358 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:33.990761 29358 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:33.990769 29358 authenticator.cpp:318] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:33.990810 29358 authenticatee.cpp:299] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:33.990828 29360 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(711)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990875 29357 master.cpp:6264] Successfully authenticated principal 'test-principal' at slave(339)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990919 29358 slave.cpp:1065] Successfully authenticated with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.990973 29358 slave.cpp:1471] Will retry registration in 63910ns if necessary [02:58:28]W: [Step 10/10] I0915 02:56:33.991029 29362 master.cpp:4886] Registering agent at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with id 43b90154-1567-471a-b415-98140670e38d-S0 [02:58:28]W: [Step 10/10] I0915 02:56:33.991137 29363 registrar.cpp:464] Applied 1 operations in 11423ns; attempting to update the registry [02:58:28]W: [Step 10/10] I0915 02:56:33.991345 29358 log.cpp:577] Attempting to append 434 bytes to the log [02:58:28]W: [Step 10/10] I0915 02:56:33.991391 29358 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 [02:58:28]W: [Step 10/10] I0915 02:56:33.991657 29364 replica.cpp:537] Replica received write request for position 3 from __req_res__(4307)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.991978 29364 leveldb.cpp:341] Persisting action (453 bytes) to leveldb took 301327ns [02:58:28]W: [Step 10/10] I0915 02:56:33.991991 29364 replica.cpp:708] Persisted action APPEND at position 3 [02:58:28]W: [Step 10/10] I0915 02:56:33.992210 29360 slave.cpp:1471] Will retry registration in 5027us if necessary [02:58:28]W: [Step 10/10] I0915 02:56:33.992264 29361 master.cpp:4874] Ignoring register agent message from slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) as admission is already in progress [02:58:28]W: [Step 10/10] I0915 02:56:33.992321 29361 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:33.992610 29361 leveldb.cpp:341] Persisting action (455 bytes) to leveldb took 273863ns [02:58:28]W: [Step 10/10] I0915 02:56:33.992625 29361 replica.cpp:708] Persisted action APPEND at position 3 [02:58:28]W: [Step 10/10] I0915 02:56:33.992933 29364 registrar.cpp:509] Successfully updated the registry in 1.776896ms [02:58:28]W: [Step 10/10] I0915 02:56:33.992955 29363 log.cpp:596] Attempting to truncate the log to 3 [02:58:28]W: [Step 10/10] I0915 02:56:33.993018 29363 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 [02:58:28]W: [Step 10/10] I0915 02:56:33.993167 29364 master.cpp:4956] Registered agent 43b90154-1567-471a-b415-98140670e38d-S0 at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] [02:58:28]W: [Step 10/10] I0915 02:56:33.993192 29357 slave.cpp:4064] Received ping from slave-observer(334)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.993223 29359 hierarchical.cpp:476] Added agent 43b90154-1567-471a-b415-98140670e38d-S0 (ip-172-30-2-175.ec2.internal.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) [02:58:28]W: [Step 10/10] I0915 02:56:33.993261 29357 slave.cpp:1111] Registered with master master@172.30.2.175:36619; given agent ID 43b90154-1567-471a-b415-98140670e38d-S0 [02:58:28]W: [Step 10/10] I0915 02:56:33.993276 29357 fetcher.cpp:86] Clearing fetcher cache [02:58:28]W: [Step 10/10] I0915 02:56:33.993351 29362 status_update_manager.cpp:184] Resuming sending status updates [02:58:28]W: [Step 10/10] I0915 02:56:33.993369 29363 replica.cpp:537] Replica received write request for position 4 from __req_res__(4308)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.993412 29359 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:33.993438 29359 hierarchical.cpp:1294] Performed allocation for agent 43b90154-1567-471a-b415-98140670e38d-S0 in 185611ns [02:58:28]W: [Step 10/10] I0915 02:56:33.993465 29357 slave.cpp:1134] Checkpointing SlaveInfo to '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedWhenUnused_JzbAWj/meta/slaves/43b90154-1567-471a-b415-98140670e38d-S0/slave.info' [02:58:28]W: [Step 10/10] I0915 02:56:33.993559 29360 master.cpp:6063] Sending 1 offers to framework 43b90154-1567-471a-b415-98140670e38d-0000 (default) at scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.993639 29357 slave.cpp:1171] Forwarding total oversubscribed resources [02:58:28]W: [Step 10/10] I0915 02:56:33.993710 29363 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 316930ns [02:58:28]W: [Step 10/10] I0915 02:56:33.993723 29363 replica.cpp:708] Persisted action TRUNCATE at position 4 [02:58:28]W: [Step 10/10] I0915 02:56:33.993718 29362 master.cpp:5340] Received update of agent 43b90154-1567-471a-b415-98140670e38d-S0 at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with total oversubscribed resources [02:58:28]W: [Step 10/10] I0915 02:56:33.993748 29360 sched.cpp:917] Scheduler::resourceOffers took 49683ns [02:58:28]W: [Step 10/10] I0915 02:56:33.993809 29360 hierarchical.cpp:540] Agent 43b90154-1567-471a-b415-98140670e38d-S0 (ip-172-30-2-175.ec2.internal.mesosphere.io) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]) [02:58:28]W: [Step 10/10] I0915 02:56:33.993860 29360 hierarchical.cpp:1675] No allocations performed [02:58:28]W: [Step 10/10] I0915 02:56:33.993875 29360 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:33.993892 29360 hierarchical.cpp:1294] Performed allocation for agent 43b90154-1567-471a-b415-98140670e38d-S0 in 50780ns [02:58:28]W: [Step 10/10] I0915 02:56:33.994009 29359 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] W0915 02:56:33.994153 29361 master.cpp:3260] Implicitly declining offers: [ 43b90154-1567-471a-b415-98140670e38d-O0 ] in ACCEPT call for framework 43b90154-1567-471a-b415-98140670e38d-0000 as the launch operation specified no tasks [02:58:28]W: [Step 10/10] I0915 02:56:33.994246 29361 master.cpp:3363] Processing ACCEPT call for offers: [ 43b90154-1567-471a-b415-98140670e38d-O0 ] on agent 43b90154-1567-471a-b415-98140670e38d-S0 at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) for framework 43b90154-1567-471a-b415-98140670e38d-0000 (default) at scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.994304 29359 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 278258ns [02:58:28]W: [Step 10/10] I0915 02:56:33.994328 29359 leveldb.cpp:399] Deleting ~2 keys from leveldb took 10182ns [02:58:28]W: [Step 10/10] I0915 02:56:33.994335 29359 replica.cpp:708] Persisted action TRUNCATE at position 4 [02:58:28]W: [Step 10/10] I0915 02:56:33.994498 29343 sched.cpp:226] Version: 1.1.0 [02:58:28]W: [Step 10/10] I0915 02:56:33.994510 29362 hierarchical.cpp:1003] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on agent 43b90154-1567-471a-b415-98140670e38d-S0 from framework 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] I0915 02:56:33.994541 29362 hierarchical.cpp:1040] Framework 43b90154-1567-471a-b415-98140670e38d-0000 filtered agent 43b90154-1567-471a-b415-98140670e38d-S0 for 5secs [02:58:28]W: [Step 10/10] I0915 02:56:33.994626 29357 sched.cpp:330] New master detected at master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.994648 29357 sched.cpp:396] Authenticating with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.994657 29357 sched.cpp:403] Using default CRAM-MD5 authenticatee [02:58:28]W: [Step 10/10] I0915 02:56:33.994743 29359 authenticatee.cpp:121] Creating new client SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:33.994839 29360 master.cpp:6234] Authenticating scheduler-2a4dc66a-4ab2-42c4-9adc-bb017eb66f46@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.994884 29362 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(712)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.994952 29360 authenticator.cpp:98] Creating new server SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:33.995009 29360 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 [02:58:28]W: [Step 10/10] I0915 02:56:33.995023 29360 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:33.995081 29358 authenticator.cpp:204] Received SASL authentication start [02:58:28]W: [Step 10/10] I0915 02:56:33.995108 29358 authenticator.cpp:326] Authentication requires more steps [02:58:28]W: [Step 10/10] I0915 02:56:33.995142 29358 authenticatee.cpp:259] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:33.995193 29358 authenticator.cpp:232] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:33.995208 29358 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [02:58:28]W: [Step 10/10] I0915 02:56:33.995219 29358 auxprop.cpp:181] Looking up auxiliary property '*userPassword' [02:58:28]W: [Step 10/10] I0915 02:56:33.995231 29358 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:33.995241 29358 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [02:58:28]W: [Step 10/10] I0915 02:56:33.995246 29358 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:33.995251 29358 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:33.995260 29358 authenticator.cpp:318] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:33.995301 29358 authenticatee.cpp:299] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:33.995316 29362 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(712)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.995337 29360 master.cpp:6264] Successfully authenticated principal 'test-principal' at scheduler-2a4dc66a-4ab2-42c4-9adc-bb017eb66f46@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.995417 29358 sched.cpp:502] Successfully authenticated with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.995435 29358 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.995466 29358 sched.cpp:853] Will retry registration in 187.331567ms if necessary [02:58:28]W: [Step 10/10] I0915 02:56:33.995501 29364 master.cpp:2429] Received SUBSCRIBE call for framework 'default' at scheduler-2a4dc66a-4ab2-42c4-9adc-bb017eb66f46@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.995519 29364 master.cpp:1891] Authorizing framework principal 'test-principal' to receive offers for role '*' [02:58:28]W: [Step 10/10] I0915 02:56:33.995594 29364 master.cpp:2505] Subscribing framework default with checkpointing disabled and capabilities [ ] [02:58:28]W: [Step 10/10] I0915 02:56:33.995690 29358 hierarchical.cpp:269] Added framework 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.995702 29364 sched.cpp:743] Framework registered with 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.995718 29364 sched.cpp:757] Scheduler::registered took 6269ns [02:58:28]W: [Step 10/10] I0915 02:56:33.995818 29358 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:33.995844 29358 hierarchical.cpp:1271] Performed allocation for 1 agents in 136137ns [02:58:28]W: [Step 10/10] I0915 02:56:33.995908 29360 master.cpp:6063] Sending 1 offers to framework 43b90154-1567-471a-b415-98140670e38d-0001 (default) at scheduler-2a4dc66a-4ab2-42c4-9adc-bb017eb66f46@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.996029 29360 sched.cpp:917] Scheduler::resourceOffers took 38207ns [02:58:28]W: [Step 10/10] I0915 02:56:33.996129 29343 sched.cpp:1987] Asked to stop the driver [02:58:28]W: [Step 10/10] I0915 02:56:33.996187 29364 sched.cpp:1187] Stopping framework 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] I0915 02:56:33.996261 29360 master.cpp:6639] Processing TEARDOWN call for framework 43b90154-1567-471a-b415-98140670e38d-0000 (default) at scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.996280 29360 master.cpp:6651] Removing framework 43b90154-1567-471a-b415-98140670e38d-0000 (default) at scheduler-3b17e9c1-a354-4d42-9e84-a32d8d28722e@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.996305 29343 sched.cpp:1987] Asked to stop the driver [02:58:28]W: [Step 10/10] I0915 02:56:33.996376 29357 hierarchical.cpp:380] Deactivated framework 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] I0915 02:56:33.996399 29362 slave.cpp:2496] Asked to shut down framework 43b90154-1567-471a-b415-98140670e38d-0000 by master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.996402 29357 hierarchical.cpp:331] Removed framework 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] W0915 02:56:33.996419 29362 slave.cpp:2511] Cannot shut down unknown framework 43b90154-1567-471a-b415-98140670e38d-0000 [02:58:28]W: [Step 10/10] I0915 02:56:33.996407 29364 sched.cpp:1187] Stopping framework 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.996501 29364 master.cpp:6639] Processing TEARDOWN call for framework 43b90154-1567-471a-b415-98140670e38d-0001 (default) at scheduler-2a4dc66a-4ab2-42c4-9adc-bb017eb66f46@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.996522 29364 master.cpp:6651] Removing framework 43b90154-1567-471a-b415-98140670e38d-0001 (default) at scheduler-2a4dc66a-4ab2-42c4-9adc-bb017eb66f46@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:33.996603 29360 hierarchical.cpp:380] Deactivated framework 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.996626 29357 slave.cpp:2496] Asked to shut down framework 43b90154-1567-471a-b415-98140670e38d-0001 by master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] W0915 02:56:33.996641 29357 slave.cpp:2511] Cannot shut down unknown framework 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.996837 29363 hierarchical.cpp:1003] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on agent 43b90154-1567-471a-b415-98140670e38d-S0 from framework 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.996971 29363 hierarchical.cpp:331] Removed framework 43b90154-1567-471a-b415-98140670e38d-0001 [02:58:28]W: [Step 10/10] I0915 02:56:33.997267 29343 slave.cpp:783] Agent terminating [02:58:28]W: [Step 10/10] I0915 02:56:33.997371 29364 master.cpp:1254] Agent 43b90154-1567-471a-b415-98140670e38d-S0 at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) disconnected [02:58:28]W: [Step 10/10] I0915 02:56:33.997386 29364 master.cpp:2789] Disconnecting agent 43b90154-1567-471a-b415-98140670e38d-S0 at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) [02:58:28]W: [Step 10/10] I0915 02:56:33.997407 29364 master.cpp:2808] Deactivating agent 43b90154-1567-471a-b415-98140670e38d-S0 at slave(339)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) [02:58:28]W: [Step 10/10] I0915 02:56:33.997628 29364 hierarchical.cpp:569] Agent 43b90154-1567-471a-b415-98140670e38d-S0 deactivated [02:58:28]W: [Step 10/10] I0915 02:56:33.998772 29343 master.cpp:1097] Master terminating [02:58:28]W: [Step 10/10] I0915 02:56:33.998905 29363 hierarchical.cpp:508] Removed agent 43b90154-1567-471a-b415-98140670e38d-S0 [02:58:28]W: [Step 10/10] I0915 02:56:34.011337 29343 cluster.cpp:157] Creating default 'local' authorizer [02:58:28]W: [Step 10/10] I0915 02:56:34.014022 29343 leveldb.cpp:174] Opened db in 2.573618ms [02:58:28]W: [Step 10/10] I0915 02:56:34.014456 29343 leveldb.cpp:181] Compacted db in 414273ns [02:58:28]W: [Step 10/10] I0915 02:56:34.014474 29343 leveldb.cpp:196] Created db iterator in 3806ns [02:58:28]W: [Step 10/10] I0915 02:56:34.014480 29343 leveldb.cpp:202] Seeked to beginning of db in 428ns [02:58:28]W: [Step 10/10] I0915 02:56:34.014485 29343 leveldb.cpp:271] Iterated through 0 keys in the db in 400ns [02:58:28]W: [Step 10/10] I0915 02:56:34.014497 29343 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned [02:58:28]W: [Step 10/10] I0915 02:56:34.014726 29359 recover.cpp:451] Starting replica recovery [02:58:28]W: [Step 10/10] I0915 02:56:34.014827 29360 recover.cpp:477] Replica is in EMPTY status [02:58:28]W: [Step 10/10] I0915 02:56:34.015197 29360 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(4309)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.015269 29357 recover.cpp:197] Received a recover response from a replica in EMPTY status [02:58:28]W: [Step 10/10] I0915 02:56:34.015357 29361 recover.cpp:568] Updating replica status to STARTING [02:58:28]W: [Step 10/10] I0915 02:56:34.015712 29357 master.cpp:380] Master e2ae91ef-64cf-49ce-90a7-daf42a34926b (ip-172-30-2-175.ec2.internal.mesosphere.io) started on 172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.015725 29357 master.cpp:382] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/COmnsK/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="100secs" --registry_strict="true" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/COmnsK/master" --zk_session_timeout="10secs" [02:58:28]W: [Step 10/10] I0915 02:56:34.015838 29357 master.cpp:432] Master only allowing authenticated frameworks to register [02:58:28]W: [Step 10/10] I0915 02:56:34.015846 29357 master.cpp:446] Master only allowing authenticated agents to register [02:58:28]W: [Step 10/10] I0915 02:56:34.015851 29357 master.cpp:459] Master only allowing authenticated HTTP frameworks to register [02:58:28]W: [Step 10/10] I0915 02:56:34.015856 29357 credentials.hpp:37] Loading credentials for authentication from '/tmp/COmnsK/credentials' [02:58:28]W: [Step 10/10] I0915 02:56:34.015931 29357 master.cpp:504] Using default 'crammd5' authenticator [02:58:28]W: [Step 10/10] I0915 02:56:34.015972 29357 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-readonly' [02:58:28]W: [Step 10/10] I0915 02:56:34.016027 29357 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-readwrite' [02:58:28]W: [Step 10/10] I0915 02:56:34.016057 29357 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-master-scheduler' [02:58:28]W: [Step 10/10] I0915 02:56:34.016086 29357 master.cpp:584] Authorization enabled [02:58:28]W: [Step 10/10] I0915 02:56:34.016140 29360 whitelist_watcher.cpp:77] No whitelist given [02:58:28]W: [Step 10/10] I0915 02:56:34.016157 29359 hierarchical.cpp:149] Initialized hierarchical allocator process [02:58:28]W: [Step 10/10] I0915 02:56:34.016180 29361 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 769257ns [02:58:28]W: [Step 10/10] I0915 02:56:34.016197 29361 replica.cpp:320] Persisted replica status to STARTING [02:58:28]W: [Step 10/10] I0915 02:56:34.016346 29364 recover.cpp:477] Replica is in STARTING status [02:58:28]W: [Step 10/10] I0915 02:56:34.016665 29359 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(4310)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.016747 29364 recover.cpp:197] Received a recover response from a replica in STARTING status [02:58:28]W: [Step 10/10] I0915 02:56:34.016875 29361 master.cpp:1855] Elected as the leading master! [02:58:28]W: [Step 10/10] I0915 02:56:34.016894 29361 master.cpp:1556] Recovering from registrar [02:58:28]W: [Step 10/10] I0915 02:56:34.016958 29360 recover.cpp:568] Updating replica status to VOTING [02:58:28]W: [Step 10/10] I0915 02:56:34.016983 29357 registrar.cpp:332] Recovering registrar [02:58:28]W: [Step 10/10] I0915 02:56:34.017462 29360 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 305086ns [02:58:28]W: [Step 10/10] I0915 02:56:34.017475 29360 replica.cpp:320] Persisted replica status to VOTING [02:58:28]W: [Step 10/10] I0915 02:56:34.017511 29360 recover.cpp:582] Successfully joined the Paxos group [02:58:28]W: [Step 10/10] I0915 02:56:34.017562 29360 recover.cpp:466] Recover process terminated [02:58:28]W: [Step 10/10] I0915 02:56:34.017699 29359 log.cpp:553] Attempting to start the writer [02:58:28]W: [Step 10/10] I0915 02:56:34.018075 29361 replica.cpp:493] Replica received implicit promise request from __req_res__(4311)@172.30.2.175:36619 with proposal 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.018365 29361 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 270298ns [02:58:28]W: [Step 10/10] I0915 02:56:34.018378 29361 replica.cpp:342] Persisted promised to 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.018579 29362 coordinator.cpp:238] Coordinator attempting to fill missing positions [02:58:28]W: [Step 10/10] I0915 02:56:34.019019 29362 replica.cpp:388] Replica received explicit promise request from __req_res__(4312)@172.30.2.175:36619 for position 0 with proposal 2 [02:58:28]W: [Step 10/10] I0915 02:56:34.019321 29362 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 279732ns [02:58:28]W: [Step 10/10] I0915 02:56:34.019335 29362 replica.cpp:708] Persisted action NOP at position 0 [02:58:28]W: [Step 10/10] I0915 02:56:34.019747 29361 replica.cpp:537] Replica received write request for position 0 from __req_res__(4313)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.019770 29361 leveldb.cpp:436] Reading position from leveldb took 8902ns [02:58:28]W: [Step 10/10] I0915 02:56:34.020041 29361 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 257735ns [02:58:28]W: [Step 10/10] I0915 02:56:34.020052 29361 replica.cpp:708] Persisted action NOP at position 0 [02:58:28]W: [Step 10/10] I0915 02:56:34.020290 29358 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:34.020586 29358 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 274885ns [02:58:28]W: [Step 10/10] I0915 02:56:34.020601 29358 replica.cpp:708] Persisted action NOP at position 0 [02:58:28]W: [Step 10/10] I0915 02:56:34.020793 29357 log.cpp:569] Writer started with ending position 0 [02:58:28]W: [Step 10/10] I0915 02:56:34.021033 29358 leveldb.cpp:436] Reading position from leveldb took 8382ns [02:58:28]W: [Step 10/10] I0915 02:56:34.021250 29358 registrar.cpp:365] Successfully fetched the registry (0B) in 4.2432ms [02:58:28]W: [Step 10/10] I0915 02:56:34.021282 29358 registrar.cpp:464] Applied 1 operations in 3830ns; attempting to update the registry [02:58:28]W: [Step 10/10] I0915 02:56:34.021468 29359 log.cpp:577] Attempting to append 235 bytes to the log [02:58:28]W: [Step 10/10] I0915 02:56:34.021533 29363 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.021796 29363 replica.cpp:537] Replica received write request for position 1 from __req_res__(4314)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.022089 29363 leveldb.cpp:341] Persisting action (254 bytes) to leveldb took 273629ns [02:58:28]W: [Step 10/10] I0915 02:56:34.022101 29363 replica.cpp:708] Persisted action APPEND at position 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.022322 29359 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:34.022608 29359 leveldb.cpp:341] Persisting action (256 bytes) to leveldb took 266817ns [02:58:28]W: [Step 10/10] I0915 02:56:34.022624 29359 replica.cpp:708] Persisted action APPEND at position 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.022864 29361 registrar.cpp:509] Successfully updated the registry in 1.56288ms [02:58:28]W: [Step 10/10] I0915 02:56:34.022925 29361 registrar.cpp:395] Successfully recovered registrar [02:58:28]W: [Step 10/10] I0915 02:56:34.022951 29362 log.cpp:596] Attempting to truncate the log to 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.023061 29361 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 [02:58:28]W: [Step 10/10] I0915 02:56:34.023100 29362 master.cpp:1664] Recovered 0 agents from the registry (196B); allowing 10mins for agents to re-register [02:58:28]W: [Step 10/10] I0915 02:56:34.023124 29364 hierarchical.cpp:176] Skipping recovery of hierarchical allocator: nothing to recover [02:58:28]W: [Step 10/10] I0915 02:56:34.023362 29364 replica.cpp:537] Replica received write request for position 2 from __req_res__(4315)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.023670 29364 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 288282ns [02:58:28]W: [Step 10/10] I0915 02:56:34.023684 29364 replica.cpp:708] Persisted action TRUNCATE at position 2 [02:58:28]W: [Step 10/10] I0915 02:56:34.023929 29358 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:34.024225 29358 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 278984ns [02:58:28]W: [Step 10/10] I0915 02:56:34.024245 29358 leveldb.cpp:399] Deleting ~1 keys from leveldb took 7851ns [02:58:28]W: [Step 10/10] I0915 02:56:34.024253 29358 replica.cpp:708] Persisted action TRUNCATE at position 2 [02:58:28]W: [Step 10/10] I0915 02:56:34.025671 29343 containerizer.cpp:217] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni [02:58:28]W: [Step 10/10] I0915 02:56:34.030243 29343 linux_launcher.cpp:104] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher [02:58:28]W: [Step 10/10] I0915 02:56:34.031205 29343 cluster.cpp:436] Creating default 'local' authorizer [02:58:28]W: [Step 10/10] I0915 02:56:34.031692 29360 slave.cpp:208] Mesos agent started on (340)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.031705 29360 slave.cpp:209] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_VuZ4TX/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_VuZ4TX/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_credentials="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_VuZ4TX/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_VuZ4TX" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_ctPdbR" --xfs_project_range="[5000-10000]" [02:58:28]W: [Step 10/10] I0915 02:56:34.031960 29360 credentials.hpp:86] Loading credential for authentication from '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_VuZ4TX/credential' [02:58:28]W: [Step 10/10] I0915 02:56:34.032016 29360 slave.cpp:346] Agent using credential for: test-principal [02:58:28]W: [Step 10/10] I0915 02:56:34.032027 29360 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_VuZ4TX/http_credentials' [02:58:28]W: [Step 10/10] I0915 02:56:34.032085 29360 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readonly' [02:58:28]W: [Step 10/10] I0915 02:56:34.032137 29360 http.cpp:883] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite' [02:58:28]W: [Step 10/10] I0915 02:56:34.032271 29343 sched.cpp:226] Version: 1.1.0 [02:58:28]W: [Step 10/10] I0915 02:56:34.032409 29360 slave.cpp:529] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] [02:58:28]W: [Step 10/10] I0915 02:56:34.032429 29364 sched.cpp:330] New master detected at master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.032438 29360 slave.cpp:537] Agent attributes: [ ] [02:58:28]W: [Step 10/10] I0915 02:56:34.032444 29360 slave.cpp:542] Agent hostname: ip-172-30-2-175.ec2.internal.mesosphere.io [02:58:28]W: [Step 10/10] I0915 02:56:34.032454 29364 sched.cpp:396] Authenticating with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.032460 29364 sched.cpp:403] Using default CRAM-MD5 authenticatee [02:58:28]W: [Step 10/10] I0915 02:56:34.032529 29358 authenticatee.cpp:121] Creating new client SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:34.032640 29357 master.cpp:6234] Authenticating scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.032692 29364 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(713)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.032788 29363 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_ctPdbR/meta' [02:58:28]W: [Step 10/10] I0915 02:56:34.032805 29364 authenticator.cpp:98] Creating new server SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:34.032871 29360 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 [02:58:28]W: [Step 10/10] I0915 02:56:34.032893 29360 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:34.032943 29363 status_update_manager.cpp:203] Recovering status update manager [02:58:28]W: [Step 10/10] I0915 02:56:34.032968 29361 authenticator.cpp:204] Received SASL authentication start [02:58:28]W: [Step 10/10] I0915 02:56:34.033013 29361 authenticator.cpp:326] Authentication requires more steps [02:58:28]W: [Step 10/10] I0915 02:56:34.033038 29359 containerizer.cpp:578] Recovering containerizer [02:58:28]W: [Step 10/10] I0915 02:56:34.033090 29361 authenticatee.cpp:259] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:34.033152 29361 authenticator.cpp:232] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:34.033174 29361 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [02:58:28]W: [Step 10/10] I0915 02:56:34.033184 29361 auxprop.cpp:181] Looking up auxiliary property '*userPassword' [02:58:28]W: [Step 10/10] I0915 02:56:34.033197 29361 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:34.033210 29361 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [02:58:28]W: [Step 10/10] I0915 02:56:34.033218 29361 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:34.033227 29361 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:34.033247 29361 authenticator.cpp:318] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:34.033309 29362 authenticatee.cpp:299] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:34.033329 29361 master.cpp:6264] Successfully authenticated principal 'test-principal' at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.033309 29357 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(713)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.033476 29363 sched.cpp:502] Successfully authenticated with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.033493 29363 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.033538 29363 sched.cpp:853] Will retry registration in 181.877426ms if necessary [02:58:28]W: [Step 10/10] I0915 02:56:34.033583 29360 master.cpp:2429] Received SUBSCRIBE call for framework 'default' at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.033608 29360 master.cpp:1891] Authorizing framework principal 'test-principal' to receive offers for role '*' [02:58:28]W: [Step 10/10] I0915 02:56:34.033715 29361 master.cpp:2505] Subscribing framework default with checkpointing disabled and capabilities [ ] [02:58:28]W: [Step 10/10] I0915 02:56:34.033861 29362 hierarchical.cpp:269] Added framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 [02:58:28]W: [Step 10/10] I0915 02:56:34.033880 29362 hierarchical.cpp:1675] No allocations performed [02:58:28]W: [Step 10/10] I0915 02:56:34.033891 29362 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:34.033902 29361 sched.cpp:743] Framework registered with e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 [02:58:28]W: [Step 10/10] I0915 02:56:34.033917 29362 hierarchical.cpp:1271] Performed allocation for 0 agents in 42607ns [02:58:28]W: [Step 10/10] I0915 02:56:34.033941 29361 sched.cpp:757] Scheduler::registered took 19437ns [02:58:28]W: [Step 10/10] I0915 02:56:34.034268 29364 provisioner.cpp:253] Provisioner recovery complete [02:58:28]W: [Step 10/10] I0915 02:56:34.034380 29362 slave.cpp:5197] Finished recovery [02:58:28]W: [Step 10/10] I0915 02:56:34.034592 29362 slave.cpp:5369] Querying resource estimator for oversubscribable resources [02:58:28]W: [Step 10/10] I0915 02:56:34.034682 29362 slave.cpp:911] New master detected at master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.034695 29362 slave.cpp:970] Authenticating with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.034698 29361 status_update_manager.cpp:177] Pausing sending status updates [02:58:28]W: [Step 10/10] I0915 02:56:34.034706 29362 slave.cpp:981] Using default CRAM-MD5 authenticatee [02:58:28]W: [Step 10/10] I0915 02:56:34.034755 29362 slave.cpp:943] Detecting new master [02:58:28]W: [Step 10/10] I0915 02:56:34.034780 29357 authenticatee.cpp:121] Creating new client SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:34.034796 29362 slave.cpp:5383] Received oversubscribable resources from the resource estimator [02:58:28]W: [Step 10/10] I0915 02:56:34.034853 29362 master.cpp:6234] Authenticating slave(340)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.034906 29360 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(714)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.034963 29360 authenticator.cpp:98] Creating new server SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:34.035023 29360 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 [02:58:28]W: [Step 10/10] I0915 02:56:34.035037 29360 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:34.035089 29363 authenticator.cpp:204] Received SASL authentication start [02:58:28]W: [Step 10/10] I0915 02:56:34.035118 29363 authenticator.cpp:326] Authentication requires more steps [02:58:28]W: [Step 10/10] I0915 02:56:34.035156 29363 authenticatee.cpp:259] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:34.035220 29364 authenticator.cpp:232] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:34.035236 29364 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [02:58:28]W: [Step 10/10] I0915 02:56:34.035243 29364 auxprop.cpp:181] Looking up auxiliary property '*userPassword' [02:58:28]W: [Step 10/10] I0915 02:56:34.035249 29364 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:34.035255 29364 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [02:58:28]W: [Step 10/10] I0915 02:56:34.035260 29364 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:34.035264 29364 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:34.035272 29364 authenticator.cpp:318] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:34.035320 29364 authenticatee.cpp:299] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:34.035336 29361 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(714)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.035363 29363 master.cpp:6264] Successfully authenticated principal 'test-principal' at slave(340)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.035467 29364 slave.cpp:1065] Successfully authenticated with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.035516 29364 slave.cpp:1471] Will retry registration in 11.557971ms if necessary [02:58:28]W: [Step 10/10] I0915 02:56:34.035564 29362 master.cpp:4886] Registering agent at slave(340)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with id e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 [02:58:28]W: [Step 10/10] I0915 02:56:34.035661 29360 registrar.cpp:464] Applied 1 operations in 12096ns; attempting to update the registry [02:58:28]W: [Step 10/10] I0915 02:56:34.035871 29359 log.cpp:577] Attempting to append 434 bytes to the log [02:58:28]W: [Step 10/10] I0915 02:56:34.035917 29364 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 [02:58:28]W: [Step 10/10] I0915 02:56:34.036151 29360 replica.cpp:537] Replica received write request for position 3 from __req_res__(4316)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.036480 29360 leveldb.cpp:341] Persisting action (453 bytes) to leveldb took 308902ns [02:58:28]W: [Step 10/10] I0915 02:56:34.036495 29360 replica.cpp:708] Persisted action APPEND at position 3 [02:58:28]W: [Step 10/10] I0915 02:56:34.036756 29362 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:34.037040 29362 leveldb.cpp:341] Persisting action (455 bytes) to leveldb took 260810ns [02:58:28]W: [Step 10/10] I0915 02:56:34.037055 29362 replica.cpp:708] Persisted action APPEND at position 3 [02:58:28]W: [Step 10/10] I0915 02:56:34.037369 29362 registrar.cpp:509] Successfully updated the registry in 1.684992ms [02:58:28]W: [Step 10/10] I0915 02:56:34.037428 29358 log.cpp:596] Attempting to truncate the log to 3 [02:58:28]W: [Step 10/10] I0915 02:56:34.037492 29362 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 [02:58:28]W: [Step 10/10] I0915 02:56:34.037580 29359 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.037677 29360 slave.cpp:1111] Registered with master master@172.30.2.175:36619; given agent ID e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 [02:58:28]W: [Step 10/10] I0915 02:56:34.037662 29357 master.cpp:4956] Registered agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 at slave(340)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] [02:58:28]W: [Step 10/10] I0915 02:56:34.037695 29360 fetcher.cpp:86] Clearing fetcher cache [02:58:28]W: [Step 10/10] I0915 02:56:34.037783 29363 status_update_manager.cpp:184] Resuming sending status updates [02:58:28]W: [Step 10/10] I0915 02:56:34.037786 29364 hierarchical.cpp:476] Added agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 (ip-172-30-2-175.ec2.internal.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: ) [02:58:28]W: [Step 10/10] I0915 02:56:34.037843 29359 replica.cpp:537] Replica received write request for position 4 from __req_res__(4317)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.037914 29360 slave.cpp:1134] Checkpointing SlaveInfo to '/mnt/teamcity/temp/buildTmp/ResourceOffersTest_ResourcesGetReofferedAfterTaskInfoError_ctPdbR/meta/slaves/e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0/slave.info' [02:58:28]W: [Step 10/10] I0915 02:56:34.037962 29364 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:34.037981 29364 hierarchical.cpp:1294] Performed allocation for agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 in 162620ns [02:58:28]W: [Step 10/10] I0915 02:56:34.038043 29360 slave.cpp:1171] Forwarding total oversubscribed resources [02:58:28]W: [Step 10/10] I0915 02:56:34.038105 29361 master.cpp:6063] Sending 1 offers to framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.038156 29359 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 288862ns [02:58:28]W: [Step 10/10] I0915 02:56:34.038168 29359 replica.cpp:708] Persisted action TRUNCATE at position 4 [02:58:28]W: [Step 10/10] I0915 02:56:34.038169 29361 master.cpp:5340] Received update of agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 at slave(340)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with total oversubscribed resources [02:58:28]W: [Step 10/10] I0915 02:56:34.038244 29358 hierarchical.cpp:540] Agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 (ip-172-30-2-175.ec2.internal.mesosphere.io) updated with oversubscribed resources (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]) [02:58:28]W: [Step 10/10] I0915 02:56:34.038270 29364 sched.cpp:917] Scheduler::resourceOffers took 41264ns [02:58:28]W: [Step 10/10] I0915 02:56:34.038295 29358 hierarchical.cpp:1675] No allocations performed [02:58:28]W: [Step 10/10] I0915 02:56:34.038307 29358 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:56:34.038323 29358 hierarchical.cpp:1294] Performed allocation for agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 in 48098ns [02:58:28]W: [Step 10/10] I0915 02:56:34.038499 29360 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 [02:58:28]W: [Step 10/10] I0915 02:56:34.038786 29364 master.cpp:3363] Processing ACCEPT call for offers: [ e2ae91ef-64cf-49ce-90a7-daf42a34926b-O0 ] on agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 at slave(340)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) for framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.038816 29364 master.cpp:2985] Authorizing framework principal 'test-principal' to launch task 1 [02:58:28]W: [Step 10/10] I0915 02:56:34.038820 29360 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 303434ns [02:58:28]W: [Step 10/10] I0915 02:56:34.038849 29360 leveldb.cpp:399] Deleting ~2 keys from leveldb took 13441ns [02:58:28]W: [Step 10/10] I0915 02:56:34.038857 29360 replica.cpp:708] Persisted action TRUNCATE at position 4 [02:58:28]W: [Step 10/10] I0915 02:56:34.039239 29362 master.cpp:5532] Sending status update TASK_ERROR for task 1 of framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 'Task uses invalid resources: Invalid resources: Resource 'cpus(*):-1' is invalid: Invalid scalar resource: value < 0' [02:58:28]W: [Step 10/10] I0915 02:56:34.039371 29360 sched.cpp:1025] Scheduler::statusUpdate took 30859ns [02:58:28]W: [Step 10/10] I0915 02:56:34.039469 29363 hierarchical.cpp:1003] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: ) on agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 from framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 [02:58:28]W: [Step 10/10] I0915 02:56:34.039504 29363 hierarchical.cpp:1040] Framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 filtered agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 for 5secs [02:58:28]W: [Step 10/10] I0915 02:56:34.039916 29343 sched.cpp:226] Version: 1.1.0 [02:58:28]W: [Step 10/10] I0915 02:56:34.040047 29361 sched.cpp:330] New master detected at master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040069 29361 sched.cpp:396] Authenticating with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040076 29361 sched.cpp:403] Using default CRAM-MD5 authenticatee [02:58:28]W: [Step 10/10] I0915 02:56:34.040143 29364 authenticatee.cpp:121] Creating new client SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:34.040210 29364 master.cpp:6234] Authenticating scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040259 29360 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(715)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040312 29363 authenticator.cpp:98] Creating new server SASL connection [02:58:28]W: [Step 10/10] I0915 02:56:34.040369 29363 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 [02:58:28]W: [Step 10/10] I0915 02:56:34.040385 29363 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:34.040431 29361 authenticator.cpp:204] Received SASL authentication start [02:58:28]W: [Step 10/10] I0915 02:56:34.040467 29361 authenticator.cpp:326] Authentication requires more steps [02:58:28]W: [Step 10/10] I0915 02:56:34.040501 29361 authenticatee.cpp:259] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:34.040555 29359 authenticator.cpp:232] Received SASL authentication step [02:58:28]W: [Step 10/10] I0915 02:56:34.040577 29359 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false [02:58:28]W: [Step 10/10] I0915 02:56:34.040585 29359 auxprop.cpp:181] Looking up auxiliary property '*userPassword' [02:58:28]W: [Step 10/10] I0915 02:56:34.040592 29359 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' [02:58:28]W: [Step 10/10] I0915 02:56:34.040598 29359 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-30-2-175.ec2.internal' server FQDN: 'ip-172-30-2-175.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true [02:58:28]W: [Step 10/10] I0915 02:56:34.040606 29359 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:34.040614 29359 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true [02:58:28]W: [Step 10/10] I0915 02:56:34.040628 29359 authenticator.cpp:318] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:34.040679 29361 authenticatee.cpp:299] Authentication success [02:58:28]W: [Step 10/10] I0915 02:56:34.040706 29358 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(715)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040735 29359 master.cpp:6264] Successfully authenticated principal 'test-principal' at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040758 29361 sched.cpp:502] Successfully authenticated with master master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040776 29361 sched.cpp:820] Sending SUBSCRIBE call to master@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:34.040807 29361 sched.cpp:853] Will retry registration in 1.269655978secs if necessary [02:58:28]W: [Step 10/10] I0915 02:56:34.040843 29363 master.cpp:2429] Received SUBSCRIBE call for framework 'default' at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:56:35.016984 29364 hierarchical.cpp:1861] Filtered offer with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 for framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 [02:58:28]W: [Step 10/10] I0915 02:56:49.035466 29362 slave.cpp:5369] Querying resource estimator for oversubscribable resources [02:58:28]W: [Step 10/10] I0915 02:58:04.042821 29357 master.cpp:249] Shutting down agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 due to health check timeout [02:58:28]W: [Step 10/10] I0915 02:58:28.001487 29363 master.cpp:1891] Authorizing framework principal 'test-principal' to receive offers for role '*' [02:58:28]W: [Step 10/10] I0915 02:58:28.001509 29364 hierarchical.cpp:1675] No allocations performed [02:58:28]W: [Step 10/10] I0915 02:58:28.001529 29364 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:58:28.001554 29364 hierarchical.cpp:1271] Performed allocation for 1 agents in 1.88307778531667mins [02:58:28]W: [Step 10/10] I0915 02:58:28.001616 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.001648 29357 master.cpp:249] Shutting down agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 due to health check timeout [02:58:28]W: [Step 10/10] I0915 02:58:28.001677 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.001929 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.002015 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.002202 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.002418 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.002508 29362 slave.cpp:4064] Received ping from slave-observer(335)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.002591 29362 slave.cpp:5383] Received oversubscribable resources from the resource estimator [02:58:28]W: [Step 10/10] I0915 02:58:28.002624 29362 slave.cpp:4969] Current disk usage 1.83%. Max allowed age: 6.171704718984306days [02:58:28]W: [Step 10/10] I0915 02:58:28.002874 29361 slave.cpp:783] Agent terminating [02:58:28]W: [Step 10/10] W0915 02:58:28.002966 29363 master.cpp:5669] Shutting down agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 at slave(340)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io) with message 'health check timed out' [02:58:28]W: [Step 10/10] I0915 02:58:28.003000 29363 master.cpp:6956] Removing agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 at slave(340)@172.30.2.175:36619 (ip-172-30-2-175.ec2.internal.mesosphere.io): health check timed out [02:58:28]W: [Step 10/10] I0915 02:58:28.003180 29361 hierarchical.cpp:508] Removed agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 [02:58:28]W: [Step 10/10] W0915 02:58:28.003226 29363 master.cpp:5662] Unable to shutdown unknown agent e2ae91ef-64cf-49ce-90a7-daf42a34926b-S0 [02:58:28]W: [Step 10/10] I0915 02:58:28.003252 29363 master.cpp:2505] Subscribing framework default with checkpointing disabled and capabilities [ ] [02:58:28]W: [Step 10/10] I0915 02:58:28.003264 29358 registrar.cpp:464] Applied 1 operations in 28124ns; attempting to update the registry [02:58:28]W: [Step 10/10] I0915 02:58:28.003371 29363 master.cpp:1293] Framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 disconnected [02:58:28]W: [Step 10/10] I0915 02:58:28.003391 29363 master.cpp:2730] Disconnecting framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003406 29363 master.cpp:2754] Deactivating framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003422 29363 master.cpp:1306] Giving framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 0ns to failover [02:58:28]W: [Step 10/10] I0915 02:58:28.003434 29361 hierarchical.cpp:269] Added framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 [02:58:28]W: [Step 10/10] I0915 02:58:28.003479 29361 hierarchical.cpp:1675] No allocations performed [02:58:28]W: [Step 10/10] I0915 02:58:28.003494 29361 hierarchical.cpp:1770] No inverse offers to send out! [02:58:28]W: [Step 10/10] I0915 02:58:28.003512 29361 hierarchical.cpp:1271] Performed allocation for 0 agents in 60826ns [02:58:28]W: [Step 10/10] I0915 02:58:28.003576 29363 master.cpp:1293] Framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 (default) at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 disconnected [02:58:28]W: [Step 10/10] I0915 02:58:28.003592 29363 master.cpp:2730] Disconnecting framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 (default) at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003599 29363 master.cpp:2754] Deactivating framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 (default) at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003612 29363 master.cpp:1306] Giving framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 (default) at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 0ns to failover [02:58:28]W: [Step 10/10] I0915 02:58:28.003613 29364 log.cpp:577] Attempting to append 237 bytes to the log [02:58:28]W: [Step 10/10] I0915 02:58:28.003628 29361 hierarchical.cpp:380] Deactivated framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 [02:58:28]W: [Step 10/10] I0915 02:58:28.003697 29361 hierarchical.cpp:380] Deactivated framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 [02:58:28]W: [Step 10/10] I0915 02:58:28.003697 29364 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 5 [02:58:28]W: [Step 10/10] I0915 02:58:28.003726 29363 master.cpp:5915] Framework failover timeout, removing framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003742 29363 master.cpp:6651] Removing framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 (default) at scheduler-073ebf05-1fe8-42f6-8073-dee5d68b86b7@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003810 29364 hierarchical.cpp:331] Removed framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0000 [02:58:28]W: [Step 10/10] I0915 02:58:28.003825 29363 master.cpp:5915] Framework failover timeout, removing framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 (default) at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.003844 29363 master.cpp:6651] Removing framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 (default) at scheduler-a96f96e9-2072-43fc-9203-e2ea4e5d7469@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.004113 29360 hierarchical.cpp:331] Removed framework e2ae91ef-64cf-49ce-90a7-daf42a34926b-0001 [02:58:28]W: [Step 10/10] I0915 02:58:28.004248 29359 replica.cpp:537] Replica received write request for position 5 from __req_res__(4318)@172.30.2.175:36619 [02:58:28]W: [Step 10/10] I0915 02:58:28.005373 29358 master.cpp:1097] Master terminating [02:58:28]W: [Step 10/10] I0915 02:58:28.134279 29359 leveldb.cpp:341] Persisting action (256 bytes) to leveldb took 129.988887ms [02:58:28]W: [Step 10/10] I0915 02:58:28.134332 29359 replica.cpp:708] Persisted action APPEND at position 5 [02:58:28] : [Step 10/10] [ FAILED ] ResourceOffersTest.ResourcesGetReofferedAfterTaskInfoError (114125 ms)
Note that this showed up at about the same time as MESOS-6166, which looks like a similar failure. Perhaps related?