[MESOS-5802] SlaveAuthorizerTest/0.ViewFlags is flaky.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: agent
• Labels: mesosphere race-condition slave
• Sprint: Mesosphere Sprint 39, Mesosphere Sprint 40
• Story Points: 2
• Assignee: Alexander Rojas
• Reporter: Jie Yu
• Shepherd: Vinod Kone
• Created: 07/Jul/16 15:53
• Updated: 04/Aug/16 16:19
• Resolved: 04/Aug/16 16:19

Description

[15:24:47] :	 [Step 10/10] [ RUN      ] SlaveAuthorizerTest/0.ViewFlags
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.025609 25322 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.030421 25322 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032060 25339 slave.cpp:205] Agent started on 335)@172.30.2.7:43076
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032078 25339 slave.cpp:206] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="true" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --http_credentials="/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C" --xfs_project_range="[5000-10000]"
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032306 25339 credentials.hpp:86] Loading credential for authentication from '/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C/credential'
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032424 25339 slave.cpp:343] Agent using credential for: test-principal
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032441 25339 credentials.hpp:37] Loading credentials for authentication from '/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C/http_credentials'
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032528 25339 slave.cpp:395] Using default 'basic' HTTP authenticator
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032754 25339 resources.cpp:572] Parsing resources as JSON failed: cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]
[15:24:47]W:	 [Step 10/10] Trying semicolon-delimited string format instead
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032838 25339 resources.cpp:572] Parsing resources as JSON failed: cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]
[15:24:47]W:	 [Step 10/10] Trying semicolon-delimited string format instead
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032968 25339 slave.cpp:594] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032994 25339 slave.cpp:602] Agent attributes: [  ]
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.032999 25339 slave.cpp:607] Agent hostname: ip-172-30-2-7.ec2.internal.mesosphere.io
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.033291 25339 process.cpp:3322] Handling HTTP event for process 'slave(335)' with path: '/slave(335)/flags'
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.033329 25343 state.cpp:57] Recovering state from '/mnt/teamcity/temp/buildTmp/SlaveAuthorizerTest_0_ViewFlags_OsJb5C/meta'
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.033576 25342 status_update_manager.cpp:200] Recovering status update manager
[15:24:47] :	 [Step 10/10] ../../src/tests/slave_authorization_tests.cpp:316: Failure
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.033604 25340 http.cpp:269] HTTP GET for /slave(335)/flags from 172.30.2.7:33866
[15:24:47] :	 [Step 10/10] Value of: (response).get().status
[15:24:47] :	 [Step 10/10]   Actual: "503 Service Unavailable"
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.033687 25340 containerizer.cpp:522] Recovering containerizer
[15:24:47] :	 [Step 10/10] Expected: OK().status
[15:24:47] :	 [Step 10/10] Which is: "200 OK"
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.034953 25340 process.cpp:3322] Handling HTTP event for process 'slave(335)' with path: '/slave(335)/state'
[15:24:47] :	 [Step 10/10] Agent has not finished recovery
[15:24:47] :	 [Step 10/10] ../../src/tests/slave_authorization_tests.cpp:320: Failure
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.035152 25343 http.cpp:269] HTTP GET for /slave(335)/state from 172.30.2.7:33868
[15:24:47] :	 [Step 10/10] parse: syntax error at line 1 near: Agent has not finished recovery
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.035768 25341 slave.cpp:841] Agent terminating
[15:24:47]W:	 [Step 10/10] I0707 15:24:47.036150 25337 provisioner.cpp:253] Provisioner recovery complete
[15:24:47] :	 [Step 10/10] [  FAILED  ] SlaveAuthorizerTest/0.ViewFlags, where TypeParam = mesos::internal::LocalAuthorizer (14 ms)

Issue Links

• duplicates: MESOS-5888 SlaveAuthorizerTest/ViewFlags is flaky