[MESOS-9923] AgentAPITest.GetStateWithNonTerminalCompletedTask is flaky - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Accepted
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: None
Labels:
- flaky
- foundations

Description

Observed in internal CI.

[ RUN      ] ContentType/AgentAPITest.GetStateWithNonTerminalCompletedTask/0
I0801 23:48:09.993656 27650 cluster.cpp:177] Creating default 'local' authorizer
I0801 23:48:09.995075 13632 master.cpp:440] Master ba9ed547-f27b-4426-96f7-d93f854c62ae (ip-172-16-10-65.ec2.internal) started on 172.16.10.65:59528
I0801 23:48:09.995096 13632 master.cpp:443] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1000secs" --allocator="hierarchical" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authentication_v0_timeout="15secs" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/AhLctl/credentials" --filter_gpu_resources="true" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_operator_event_stream_subscribers="1000" --max_unreachable_tasks_per_framework="1000" --memory_profiling="false" --min_allocatable_resources="cpus:0.01|mem:32" --port="5050" --publish_per_framework_metrics="true" --quiet="false" --recovery_agent_removal_limit="100%" --registry="in_memory" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --require_agent_domain="false" --role_sorter="drf" --root_submissions="true" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/AhLctl/master" --zk_session_timeout="10secs"
I0801 23:48:09.995235 13632 master.cpp:492] Master only allowing authenticated frameworks to register
I0801 23:48:09.995242 13632 master.cpp:498] Master only allowing authenticated agents to register
I0801 23:48:09.995247 13632 master.cpp:504] Master only allowing authenticated HTTP frameworks to register
I0801 23:48:09.995252 13632 credentials.hpp:37] Loading credentials for authentication from '/tmp/AhLctl/credentials'
I0801 23:48:09.995324 13632 master.cpp:548] Using default 'crammd5' authenticator
I0801 23:48:09.995362 13632 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
I0801 23:48:09.995391 13632 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
I0801 23:48:09.995409 13632 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
I0801 23:48:09.995426 13632 master.cpp:629] Authorization enabled
I0801 23:48:09.995962 13632 hierarchical.cpp:241] Initialized hierarchical allocator process
I0801 23:48:09.995999 13632 whitelist_watcher.cpp:77] No whitelist given
I0801 23:48:09.996352 13632 master.cpp:2168] Elected as the leading master!
I0801 23:48:09.996364 13632 master.cpp:1664] Recovering from registrar
I0801 23:48:09.996434 13636 registrar.cpp:339] Recovering registrar
I0801 23:48:09.996553 13636 registrar.cpp:383] Successfully fetched the registry (0B) in 103168ns
I0801 23:48:09.996584 13636 registrar.cpp:487] Applied 1 operations in 6776ns; attempting to update the registry
I0801 23:48:09.996692 13636 registrar.cpp:544] Successfully updated the registry in 94976ns
I0801 23:48:09.996717 13636 registrar.cpp:416] Successfully recovered registrar
I0801 23:48:09.996929 13632 master.cpp:1817] Recovered 0 agents from the registry (180B); allowing 10mins for agents to reregister
I0801 23:48:09.996948 13636 hierarchical.cpp:280] Skipping recovery of hierarchical allocator: nothing to recover
W0801 23:48:09.998728 27650 process.cpp:2877] Attempted to spawn already running process files@172.16.10.65:59528
I0801 23:48:09.999023 27650 cluster.cpp:518] Creating default 'local' authorizer
W0801 23:48:09.999817 27650 process.cpp:2877] Attempted to spawn already running process version@172.16.10.65:59528
I0801 23:48:09.999938 13634 slave.cpp:267] Mesos agent started on (1122)@172.16.10.65:59528
I0801 23:48:10.000281 13634 slave.cpp:268] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/AhLctl/jeVIQy/store/appc" --authenticate_http_executors="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authentication_timeout_max="1mins" --authentication_timeout_min="5secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_destroy_timeout="1mins" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/AhLctl/jeVIQy/credential" --default_role="*" --disallow_sharing_agent_ipc_namespace="false" --disallow_sharing_agent_pid_namespace="false" --disk_watch_interval="1mins" --docker="docker" --docker_ignore_runtime="false" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/AhLctl/jeVIQy/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_reregistration_timeout="2secs" --executor_shutdown_grace_period="0ns" --fetcher_cache_dir="/tmp/AhLctl/jeVIQy/fetch" --fetcher_cache_size="2GB" --fetcher_stall_timeout="1mins" --frameworks_home="/tmp/AhLctl/jeVIQy/frameworks" --gc_delay="1weeks" --gc_disk_headroom="0.1" --gc_non_executor_container_sandboxes="false" --help="false" --hostname_lookup="true" --http_command_executor="false" --http_credentials="/tmp/AhLctl/jeVIQy/http_credentials" --http_heartbeat_interval="30secs" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --jwt_secret_key="/tmp/AhLctl/jeVIQy/jwt_secret_key" --launcher="linux" --launcher_dir="/home/ubuntu/workspace/mesos/Mesos_CI-build/FLAG/SSL/label/mesos-ec2-ubuntu-14.04/mesos/build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --memory_profiling="false" --network_cni_metrics="true" --network_cni_root_dir_persist="false" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --reconfiguration_policy="equal" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_z31x3O" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5" --zk_session_timeout="10secs"
I0801 23:48:10.000499 13634 credentials.hpp:86] Loading credential for authentication from '/tmp/AhLctl/jeVIQy/credential'
I0801 23:48:10.000691 13634 slave.cpp:300] Agent using credential for: test-principal
I0801 23:48:10.000826 13634 credentials.hpp:37] Loading credentials for authentication from '/tmp/AhLctl/jeVIQy/http_credentials'
I0801 23:48:10.001021 13634 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-executor'
I0801 23:48:10.001186 13634 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-executor'
I0801 23:48:10.001374 13634 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
I0801 23:48:10.001539 13634 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readonly'
I0801 23:48:10.001704 13634 http.cpp:975] Creating default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
I0801 23:48:10.001917 13634 http.cpp:996] Creating default 'jwt' HTTP authenticator for realm 'mesos-agent-readwrite'
I0801 23:48:10.002270 13634 disk_profile_adaptor.cpp:78] Creating default disk profile adaptor module
I0801 23:48:10.002600 27650 sched.cpp:239] Version: 1.9.0
I0801 23:48:10.002848 13638 sched.cpp:343] New master detected at master@172.16.10.65:59528
I0801 23:48:10.002883 13638 sched.cpp:408] Authenticating with master master@172.16.10.65:59528
I0801 23:48:10.002892 13638 sched.cpp:415] Using default CRAM-MD5 authenticatee
I0801 23:48:10.003152 13637 authenticatee.cpp:121] Creating new client SASL connection
I0801 23:48:10.003396 13634 slave.cpp:615] Agent resources: [{"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}]
I0801 23:48:10.003455 13634 slave.cpp:623] Agent attributes: [  ]
I0801 23:48:10.003463 13634 slave.cpp:632] Agent hostname: ip-172-16-10-65.ec2.internal
I0801 23:48:10.003760 13637 master.cpp:10578] Authenticating scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:10.003895 13637 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1977)@172.16.10.65:59528
I0801 23:48:10.004032 13635 authenticator.cpp:98] Creating new server SASL connection
I0801 23:48:10.004314 13633 task_status_update_manager.cpp:181] Pausing sending task status updates
I0801 23:48:10.004338 13633 status_update_manager_process.hpp:379] Pausing operation status update manager
I0801 23:48:10.004393 13631 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0801 23:48:10.004406 13631 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0801 23:48:10.004542 13634 authenticator.cpp:204] Received SASL authentication start
I0801 23:48:10.004583 13634 authenticator.cpp:326] Authentication requires more steps
I0801 23:48:10.004588 13632 state.cpp:67] Recovering state from '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta'
I0801 23:48:10.004633 13635 slave.cpp:7444] Finished recovering checkpointed state from '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta', beginning agent recovery
I0801 23:48:10.004691 13635 task_status_update_manager.cpp:207] Recovering task status update manager
I0801 23:48:10.004880 13635 composing.cpp:339] Finished recovering all containerizers
I0801 23:48:10.004941 13636 slave.cpp:7908] Recovering executors
I0801 23:48:10.004967 13636 slave.cpp:8061] Finished recovery
I0801 23:48:10.005252 13633 task_status_update_manager.cpp:181] Pausing sending task status updates
I0801 23:48:10.005275 13633 status_update_manager_process.hpp:379] Pausing operation status update manager
I0801 23:48:10.005252 13636 slave.cpp:1351] New master detected at master@172.16.10.65:59528
I0801 23:48:10.005313 13636 slave.cpp:1416] Detecting new master
I0801 23:48:10.004611 13634 authenticatee.cpp:259] Received SASL authentication step
I0801 23:48:10.005597 13633 authenticator.cpp:232] Received SASL authentication step
I0801 23:48:10.005614 13633 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-65.ec2.internal' server FQDN: 'ip-172-16-10-65.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 
I0801 23:48:10.005621 13633 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0801 23:48:10.005631 13633 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0801 23:48:10.005640 13633 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-65.ec2.internal' server FQDN: 'ip-172-16-10-65.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 
I0801 23:48:10.005646 13633 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0801 23:48:10.005652 13633 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0801 23:48:10.005664 13633 authenticator.cpp:318] Authentication success
I0801 23:48:10.005718 13633 master.cpp:10610] Successfully authenticated principal 'test-principal' at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:10.005738 13633 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1977)@172.16.10.65:59528
I0801 23:48:10.007694 13632 slave.cpp:1443] Authenticating with master master@172.16.10.65:59528
I0801 23:48:10.007717 13632 slave.cpp:1452] Using default CRAM-MD5 authenticatee
I0801 23:48:10.011816 13634 authenticatee.cpp:299] Authentication success
I0801 23:48:10.011960 13632 authenticatee.cpp:121] Creating new client SASL connection
I0801 23:48:10.012117 13638 sched.cpp:520] Successfully authenticated with master master@172.16.10.65:59528
I0801 23:48:10.012130 13638 sched.cpp:835] Sending SUBSCRIBE call to master@172.16.10.65:59528
I0801 23:48:10.012174 13638 sched.cpp:870] Will retry registration in 1.917744521secs if necessary
I0801 23:48:10.012262 13635 master.cpp:2908] Received SUBSCRIBE call for framework 'default' at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:10.012281 13635 master.cpp:2240] Authorizing framework principal 'test-principal' to receive offers for roles '{ * }'
I0801 23:48:10.012362 13635 master.cpp:10578] Authenticating slave(1122)@172.16.10.65:59528
I0801 23:48:10.012420 13635 master.cpp:2995] Subscribing framework default with checkpointing enabled and capabilities [ MULTI_ROLE, RESERVATION_REFINEMENT ]
I0801 23:48:10.012434 13636 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(1978)@172.16.10.65:59528
I0801 23:48:10.012485 13636 authenticator.cpp:98] Creating new server SASL connection
I0801 23:48:10.012857 13635 master.cpp:10808] Adding framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 with roles {  } suppressed
I0801 23:48:10.012868 13634 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
I0801 23:48:10.012878 13634 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
I0801 23:48:10.012903 13634 authenticator.cpp:204] Received SASL authentication start
I0801 23:48:10.012931 13634 authenticator.cpp:326] Authentication requires more steps
I0801 23:48:10.012974 13634 sched.cpp:751] Framework registered with ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.012996 13634 sched.cpp:770] Scheduler::registered took 12016ns
I0801 23:48:10.013003 13635 hierarchical.cpp:368] Added framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.013015 13634 authenticatee.cpp:259] Received SASL authentication step
I0801 23:48:10.013042 13634 authenticator.cpp:232] Received SASL authentication step
I0801 23:48:10.013047 13635 hierarchical.cpp:1508] Performed allocation for 0 agents in 15293ns
I0801 23:48:10.013053 13634 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-65.ec2.internal' server FQDN: 'ip-172-16-10-65.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false 
I0801 23:48:10.013059 13634 auxprop.cpp:181] Looking up auxiliary property '*userPassword'
I0801 23:48:10.013068 13634 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
I0801 23:48:10.013075 13634 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: 'ip-172-16-10-65.ec2.internal' server FQDN: 'ip-172-16-10-65.ec2.internal' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true 
I0801 23:48:10.013082 13634 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
I0801 23:48:10.013087 13634 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true
I0801 23:48:10.013098 13634 authenticator.cpp:318] Authentication success
I0801 23:48:10.013132 13634 authenticatee.cpp:299] Authentication success
I0801 23:48:10.013139 13638 master.cpp:10610] Successfully authenticated principal 'test-principal' at slave(1122)@172.16.10.65:59528
I0801 23:48:10.013161 13638 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(1978)@172.16.10.65:59528
I0801 23:48:10.013686 13634 slave.cpp:1543] Successfully authenticated with master master@172.16.10.65:59528
I0801 23:48:10.013890 13631 master.cpp:7086] Received register agent message from slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.013960 13631 master.cpp:4202] Authorizing agent providing resources 'cpus:2; mem:1024; disk:1024; ports:[31000-32000]' with principal 'test-principal'
I0801 23:48:10.014089 13631 master.cpp:7153] Authorized registration of agent at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.014118 13631 master.cpp:7265] Registering agent at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal) with id ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.014231 13631 registrar.cpp:487] Applied 1 operations in 35429ns; attempting to update the registry
I0801 23:48:10.014349 13631 registrar.cpp:544] Successfully updated the registry in 101888ns
I0801 23:48:10.014397 13631 master.cpp:7313] Admitted agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.014499 13631 master.cpp:7358] Registered agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal) with cpus:2; mem:1024; disk:1024; ports:[31000-32000]
I0801 23:48:10.014581 13631 hierarchical.cpp:617] Added agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 (ip-172-16-10-65.ec2.internal) with cpus:2; mem:1024; disk:1024; ports:[31000-32000] (allocated: {})
I0801 23:48:10.014814 13632 master.cpp:10393] Sending offers [ ba9ed547-f27b-4426-96f7-d93f854c62ae-O0 ] to framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:10.014950 13632 sched.cpp:934] Scheduler::resourceOffers took 22407ns
I0801 23:48:10.015086 13631 hierarchical.cpp:1508] Performed allocation for 1 agents in 464279ns
I0801 23:48:10.015185 13634 slave.cpp:1993] Will retry registration in 19.857824ms if necessary
I0801 23:48:10.015347 13634 slave.cpp:1576] Registered with master master@172.16.10.65:59528; given agent ID ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.015393 13632 task_status_update_manager.cpp:188] Resuming sending task status updates
I0801 23:48:10.015497 13634 slave.cpp:1611] Checkpointing SlaveInfo to '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/slave.info'
I0801 23:48:10.015516 13638 status_update_manager_process.hpp:385] Resuming operation status update manager
I0801 23:48:10.016136 13634 slave.cpp:1663] Forwarding agent update {"operations":{},"resource_providers":{},"resource_version_uuid":{"value":"izXdb7yjTI+Jt8xfgvVR9Q=="},"slave_id":{"value":"ba9ed547-f27b-4426-96f7-d93f854c62ae-S0"},"update_oversubscribed_resources":false}
I0801 23:48:10.016438 13631 master.cpp:8457] Ignoring update on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal) as it reports no changes
I0801 23:48:10.016716 13634 master.cpp:12685] Removing offer ba9ed547-f27b-4426-96f7-d93f854c62ae-O0
I0801 23:48:10.016805 13634 master.cpp:4739] Processing ACCEPT call for offers: [ ba9ed547-f27b-4426-96f7-d93f854c62ae-O0 ] on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal) for framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:10.016844 13634 master.cpp:3756] Authorizing framework principal 'test-principal' to launch task 7ba022e2-a921-445c-90a6-1133bfec7e34
W0801 23:48:10.017196 13637 validation.cpp:1640] Executor 'default' for task '7ba022e2-a921-445c-90a6-1133bfec7e34' uses less CPUs (None) than the minimum required (0.01). Please update your executor, as this will be mandatory in future releases.
W0801 23:48:10.017215 13637 validation.cpp:1652] Executor 'default' for task '7ba022e2-a921-445c-90a6-1133bfec7e34' uses less memory (None) than the minimum required (32MB). Please update your executor, as this will be mandatory in future releases.
I0801 23:48:10.017282 13637 master.cpp:4274] Adding executor 'default' with resources {} of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.017313 13637 master.cpp:4300] Adding task 7ba022e2-a921-445c-90a6-1133bfec7e34 with resources cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.017408 13637 master.cpp:5718] Launching task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 with resources [{"allocation_info":{"role":"*"},"name":"cpus","scalar":{"value":2.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"mem","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"disk","scalar":{"value":1024.0},"type":"SCALAR"},{"allocation_info":{"role":"*"},"name":"ports","ranges":{"range":[{"begin":31000,"end":32000}]},"type":"RANGES"}] on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal) on  new executor
I0801 23:48:10.017588 13638 hierarchical.cpp:1432] Allocation paused
I0801 23:48:10.017607 13638 hierarchical.cpp:1442] Allocation resumed
I0801 23:48:10.017680 13637 slave.cpp:2130] Got assigned task '7ba022e2-a921-445c-90a6-1133bfec7e34' for framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.017745 13637 slave.cpp:9852] Checkpointing FrameworkInfo to '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/framework.info'
I0801 23:48:10.017951 13637 slave.cpp:9863] Checkpointing framework pid 'scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528' to '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/framework.pid'
I0801 23:48:10.019289 13637 slave.cpp:2504] Authorizing task '7ba022e2-a921-445c-90a6-1133bfec7e34' for framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.019320 13637 slave.cpp:9400] Authorizing framework principal 'test-principal' to launch task 7ba022e2-a921-445c-90a6-1133bfec7e34
I0801 23:48:10.019901 13637 slave.cpp:2977] Launching task '7ba022e2-a921-445c-90a6-1133bfec7e34' for framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.019950 13637 paths.cpp:810] Creating sandbox '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792' for user 'root'
I0801 23:48:10.020830 13637 slave.cpp:10650] Checkpointing ExecutorInfo to '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/executor.info'
I0801 23:48:10.021227 13637 paths.cpp:813] Creating sandbox '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792'
I0801 23:48:10.021468 13637 slave.cpp:9938] Launching executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 with resources [] in work directory '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792'
I0801 23:48:10.021822 13637 slave.cpp:10681] Checkpointing TaskInfo to '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792/tasks/7ba022e2-a921-445c-90a6-1133bfec7e34/task.info'
I0801 23:48:10.023365 13637 slave.cpp:3203] Queued task '7ba022e2-a921-445c-90a6-1133bfec7e34' for executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.023617 13637 slave.cpp:1084] Successfully attached '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792' to virtual path '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/latest'
I0801 23:48:10.023661 13637 slave.cpp:1084] Successfully attached '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792' to virtual path '/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/latest'
I0801 23:48:10.023674 13637 slave.cpp:1084] Successfully attached '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792' to virtual path '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792'
I0801 23:48:10.023810 13637 slave.cpp:3651] Launching container 7b299798-f257-40fc-8586-9b3653116792 for executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
W0801 23:48:10.024217 13637 process.cpp:2877] Attempted to spawn already running process version@172.16.10.65:59528
I0801 23:48:10.024363 13637 exec.cpp:164] Version: 1.9.0
I0801 23:48:10.024410 13632 exec.cpp:213] Executor started at: executor(204)@172.16.10.65:59528 with pid 27650
I0801 23:48:10.024456 13632 slave.cpp:5222] Got registration for executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 from executor(204)@172.16.10.65:59528
I0801 23:48:10.024495 13632 slave.cpp:5308] Checkpointing executor pid 'executor(204)@172.16.10.65:59528' to '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792/pids/libprocess.pid'
I0801 23:48:10.025211 13635 exec.cpp:237] Executor registered on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.025260 13635 exec.cpp:249] Executor::registered took 27772ns
I0801 23:48:10.025485 13637 slave.cpp:3421] Sending queued task '7ba022e2-a921-445c-90a6-1133bfec7e34' to executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 at executor(204)@172.16.10.65:59528
I0801 23:48:10.025606 13637 exec.cpp:331] Executor asked to run task '7ba022e2-a921-445c-90a6-1133bfec7e34'
I0801 23:48:10.025635 13637 exec.cpp:340] Executor::launchTask took 16382ns
I0801 23:48:10.025668 13637 exec.cpp:595] Executor sending status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.025749 13632 slave.cpp:5703] Handling status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 from executor(204)@172.16.10.65:59528
I0801 23:48:10.025915 13631 task_status_update_manager.cpp:328] Received task status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.025933 13631 task_status_update_manager.cpp:507] Creating StatusUpdate stream for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026168 13631 task_status_update_manager.cpp:842] Checkpointing UPDATE for task status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026237 13631 task_status_update_manager.cpp:383] Forwarding task status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 to the agent
I0801 23:48:10.026294 13631 slave.cpp:6229] Forwarding the update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 to master@172.16.10.65:59528
I0801 23:48:10.026373 13638 master.cpp:8946] Status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 from agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.026391 13638 master.cpp:9003] Forwarding status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026437 13638 master.cpp:12034] Updating the state of task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)
I0801 23:48:10.026468 13631 slave.cpp:6122] Task status update manager successfully handled status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026484 13631 slave.cpp:6138] Sending acknowledgement for status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 to executor(204)@172.16.10.65:59528
I0801 23:48:10.026504 13633 sched.cpp:1042] Scheduler::statusUpdate took 20738ns
I0801 23:48:10.026595 13638 master.cpp:6693] Processing ACKNOWLEDGE call for status 8c67d57a-2391-4852-b3c7-cdb20efbd303 for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.026664 13638 slave.cpp:965] Unregistering and shutting down
I0801 23:48:10.026677 13638 slave.cpp:4056] Asked to shut down framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 by @0.0.0.0:0
I0801 23:48:10.026684 13638 slave.cpp:4081] Shutting down framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026690 13638 slave.cpp:7175] Shutting down executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 at executor(204)@172.16.10.65:59528
I0801 23:48:10.026729 13638 task_status_update_manager.cpp:401] Received task status update acknowledgement (UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026751 13638 task_status_update_manager.cpp:842] Checkpointing ACK for task status update TASK_RUNNING (Status UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026805 13638 master.cpp:11599] Removing agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal): the agent unregistered
I0801 23:48:10.026836 13638 slave.cpp:7248] Killing executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 at executor(204)@172.16.10.65:59528
I0801 23:48:10.026850 13638 slave.cpp:4832] Task status update manager successfully handled status update acknowledgement (UUID: 8c67d57a-2391-4852-b3c7-cdb20efbd303) for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.026890 13638 registrar.cpp:487] Applied 1 operations in 15225ns; attempting to update the registry
I0801 23:48:10.026993 13631 exec.cpp:412] Executor received status update acknowledgement 8c67d57a-2391-4852-b3c7-cdb20efbd303 for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.027046 13631 registrar.cpp:544] Successfully updated the registry in 144128ns
I0801 23:48:10.027114 13631 slave.cpp:6409] Got exited event for executor(204)@172.16.10.65:59528
I0801 23:48:10.027144 13631 slave.cpp:6809] Executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 exited with status 0
I0801 23:48:10.027161 13638 master.cpp:11641] Removed agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal): the agent unregistered
I0801 23:48:10.027170 13631 slave.cpp:6920] Cleaning up executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 at executor(204)@172.16.10.65:59528
I0801 23:48:10.027194 13638 master.cpp:12034] Updating the state of task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (latest state: TASK_LOST, status update state: TASK_LOST)
I0801 23:48:10.027281 13638 master.cpp:12132] Removing task 7ba022e2-a921-445c-90a6-1133bfec7e34 with resources cpus(allocated: *):2; mem(allocated: *):1024; disk(allocated: *):1024; ports(allocated: *):[31000-32000] of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.027335 13638 master.cpp:8998] Sending status update TASK_LOST for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 'Agent ip-172-16-10-65.ec2.internal removed: the agent unregistered'
I0801 23:48:10.027374 13638 master.cpp:12172] Removing executor 'default' with resources {} of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 on agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 at slave(1122)@172.16.10.65:59528 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.027420 13631 slave.cpp:7049] Cleaning up framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.027460 13638 master.cpp:2085] Notifying framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 of lost agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0 (ip-172-16-10-65.ec2.internal)
I0801 23:48:10.027485 13631 slave.cpp:924] Agent terminating
W0801 23:48:10.027509 13638 master.cpp:9255] Ignoring exited executor 'default' of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 on removed agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.027611 13632 gc.cpp:95] Scheduling '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792' for gc 6.9999996843763days in the future
I0801 23:48:10.027644 13638 hierarchical.cpp:775] Removed all filters for agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.027649 13632 gc.cpp:95] Scheduling '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default' for gc 6.99999968364444days in the future
I0801 23:48:10.027654 13638 hierarchical.cpp:650] Removed agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.027673 13632 gc.cpp:95] Scheduling '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default/runs/7b299798-f257-40fc-8586-9b3653116792' for gc 6.99999968328593days in the future
I0801 23:48:10.027698 13632 gc.cpp:95] Scheduling '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000/executors/default' for gc 6.99999968302222days in the future
I0801 23:48:10.027721 13632 gc.cpp:95] Scheduling '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000' for gc 6.99999968223407days in the future
I0801 23:48:10.027798 13632 gc.cpp:95] Scheduling '/tmp/ContentType_AgentAPITest_GetStateWithNonTerminalCompletedTask_0_nNgng5/meta/slaves/ba9ed547-f27b-4426-96f7-d93f854c62ae-S0/frameworks/ba9ed547-f27b-4426-96f7-d93f854c62ae-0000' for gc 6.99999968202667days in the future
I0801 23:48:10.027812 13638 sched.cpp:1042] Scheduler::statusUpdate took 9674ns
I0801 23:48:10.027824 13632 task_status_update_manager.cpp:289] Closing task status update streams for framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.027833 13632 task_status_update_manager.cpp:538] Cleaning up status update stream for task 7ba022e2-a921-445c-90a6-1133bfec7e34 of framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:10.027840 13638 sched.cpp:1104] Lost agent ba9ed547-f27b-4426-96f7-d93f854c62ae-S0
I0801 23:48:10.027856 13638 sched.cpp:1115] Scheduler::slaveLost took 7643ns
../../src/tests/api_tests.cpp:7085: Failure
Failed to wait 15secs for shutdown
I0801 23:48:25.029498 13635 master.cpp:1410] Framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 disconnected
I0801 23:48:25.029528 13635 master.cpp:3360] Deactivating framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:25.029554 13635 master.cpp:3337] Disconnecting framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:25.029567 13635 master.cpp:1425] Giving framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528 0ns to failover
I0801 23:48:25.029685 13637 hierarchical.cpp:475] Deactivated framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:25.029685 13636 master.cpp:10185] Framework failover timeout, removing framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:25.029711 13636 master.cpp:11184] Removing framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000 (default) at scheduler-578a9bfc-813a-4924-ada5-f0f6431f4b0c@172.16.10.65:59528
I0801 23:48:25.029824 13636 hierarchical.cpp:1432] Allocation paused
I0801 23:48:25.029880 13636 hierarchical.cpp:417] Removed framework ba9ed547-f27b-4426-96f7-d93f854c62ae-0000
I0801 23:48:25.029903 13636 hierarchical.cpp:1442] Allocation resumed
../../src/tests/api_tests.cpp:7080: Failure
Actual function call count doesn't match EXPECT_CALL(exec, shutdown(_))...
         Expected: to be called once
           Actual: never called - unsatisfied and active
I0801 23:48:25.032428 27650 master.cpp:1135] Master terminating
[  FAILED  ] ContentType/AgentAPITest.GetStateWithNonTerminalCompletedTask/0, where GetParam() = application/x-protobuf (15051 ms)

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Andrei Sekretenko

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 02/Aug/19 13:56

Updated:: 29/Aug/19 16:02