Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
None
-
None
-
None
-
None
-
macOS 10.13.2 (17C88)
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Description
The problem appears to cause several switchboard tests to fail. Note that this problem does not manifest on older Apple systems.
The failure rate on this system is 100%.
List of currently failing tests:
IOSwitchboardTest.ContainerAttach IOSwitchboardTest.ContainerAttachAfterSlaveRestart IOSwitchboardTest.OutputRedirectionWithTTY ContentType/AgentAPITest.LaunchNestedContainerSessionWithTTY/0 ContentType/AgentAPITest.LaunchNestedContainerSessionWithTTY/1
This is an example using GLOG=v1 verbose logging:
[ RUN ] IOSwitchboardTest.ContainerAttach
I0201 03:02:51.925930 2385417024 containerizer.cpp:304] Using isolation { environment_secret, filesystem/posix, posix/cpu }
I0201 03:02:51.926230 2385417024 provisioner.cpp:299] Using default backend 'copy'
I0201 03:02:51.927325 107409408 containerizer.cpp:674] Recovering containerizer
I0201 03:02:51.928336 109019136 provisioner.cpp:495] Provisioner recovery complete
I0201 03:02:51.934250 105799680 containerizer.cpp:1202] Starting container 1b1af888-9e39-4c13-a647-ac43c0df9fad
I0201 03:02:51.936218 105799680 containerizer.cpp:1368] Checkpointed ContainerConfig at '/var/folders/_t/rdp354gx7j5fjww270kbk6_r0000gn/T/IOSwitchboardTest_ContainerAttach_1nkPYl/containers/1b1af888-9e39-4c13-a647-ac43c0df9fad/config'
I0201 03:02:51.936251 105799680 containerizer.cpp:2952] Transitioning the state of container 1b1af888-9e39-4c13-a647-ac43c0df9fad from PROVISIONING to PREPARING
I0201 03:02:51.937369 109019136 switchboard.cpp:429] Allocated pseudo terminal '/dev/ttys003' for container 1b1af888-9e39-4c13-a647-ac43c0df9fad
I0201 03:02:51.943632 109019136 switchboard.cpp:557] Launching 'mesos-io-switchboard' with flags '--heartbeat_interval="30secs" --help="false" --socket_address="/tmp/mesos-io-switchboard-d3bcec3f-7c29-4630-b374-55fabb6034d8" --stderr_from_fd="7" --stderr_to_fd="2" --stdin_to_fd="7" --stdout_from_fd="7" --stdout_to_fd="1" --tty="true" --wait_for_connection="false"' for container 1b1af888-9e39-4c13-a647-ac43c0df9fad
I0201 03:02:51.945106 109019136 switchboard.cpp:587] Created I/O switchboard server (pid: 83716) listening on socket file '/tmp/mesos-io-switchboard-d3bcec3f-7c29-4630-b374-55fabb6034d8' for container 1b1af888-9e39-4c13-a647-ac43c0df9fad
I0201 03:02:51.947762 106336256 containerizer.cpp:1844] Launching 'mesos-containerizer' with flags '--help="false" --launch_info="{"command":{"shell":true,"value":"sleep 1000"},"environment":{"variables":[{"name":"MESOS_SANDBOX","type":"VALUE","value":"\/var\/folders\/_t\/rdp354gx7j5fjww270kbk6_r0000gn\/T\/IOSwitchboardTest_ContainerAttach_W9gDw0"}]},"task_environment":{},"tty_slave_path":"\/dev\/ttys003","working_directory":"\/var\/folders\/_t\/rdp354gx7j5fjww270kbk6_r0000gn\/T\/IOSwitchboardTest_ContainerAttach_W9gDw0"}" --pipe_read="7" --pipe_write="10" --runtime_directory="/var/folders/_t/rdp354gx7j5fjww270kbk6_r0000gn/T/IOSwitchboardTest_ContainerAttach_1nkPYl/containers/1b1af888-9e39-4c13-a647-ac43c0df9fad"'
I0201 03:02:51.949144 106336256 launcher.cpp:140] Forked child with pid '83717' for container '1b1af888-9e39-4c13-a647-ac43c0df9fad'
I0201 03:02:51.949896 106336256 containerizer.cpp:2952] Transitioning the state of container 1b1af888-9e39-4c13-a647-ac43c0df9fad from PREPARING to ISOLATING
I0201 03:02:51.951071 106336256 containerizer.cpp:2952] Transitioning the state of container 1b1af888-9e39-4c13-a647-ac43c0df9fad from ISOLATING to FETCHING
I0201 03:02:51.951190 108482560 fetcher.cpp:369] Starting to fetch URIs for container: 1b1af888-9e39-4c13-a647-ac43c0df9fad, directory: /var/folders/_t/rdp354gx7j5fjww270kbk6_r0000gn/T/IOSwitchboardTest_ContainerAttach_W9gDw0
I0201 03:02:51.951791 109019136 containerizer.cpp:2952] Transitioning the state of container 1b1af888-9e39-4c13-a647-ac43c0df9fad from FETCHING to RUNNING
I0201 03:02:52.076602 106872832 containerizer.cpp:2338] Destroying container 1b1af888-9e39-4c13-a647-ac43c0df9fad in RUNNING state
I0201 03:02:52.076644 106872832 containerizer.cpp:2952] Transitioning the state of container 1b1af888-9e39-4c13-a647-ac43c0df9fad from RUNNING to DESTROYING
I0201 03:02:52.076920 106872832 launcher.cpp:156] Asked to destroy container 1b1af888-9e39-4c13-a647-ac43c0df9fad
I0201 03:02:52.158571 107945984 containerizer.cpp:2791] Container 1b1af888-9e39-4c13-a647-ac43c0df9fad has exited
I0201 03:02:57.162788 110092288 switchboard.cpp:790] Sending SIGTERM to I/O switchboard server (pid: 83716) since container 1b1af888-9e39-4c13-a647-ac43c0df9fad is being destroyed
../../src/tests/containerizer/io_switchboard_tests.cpp:760: Failure
Failed to wait 15secs for destroy
[ FAILED ] IOSwitchboardTest.ContainerAttach (15156 ms)
[----------] 1 test from IOSwitchboardTest (15158 ms total)
[----------] Global test environment tear-down
../../src/tests/environment.cpp:832: Failure
Failed
Tests completed with child processes remaining:
-+- 83701 /Users/till/Development/mesos-private/build/src/.libs/mesos-tests --gtest_filter=IOSwitchboardTest.ContainerAttach --verbose
\--- 83716 /Users/till/Development/mesos-private/build/src/.libs/mesos-io-switchboard --heartbeat_interval=30secs --help=false --socket_address=/tmp/mesos-io-switchboard-d3bcec3f-7c29-4630-b374-55fabb6034d8 --stderr_from_fd=7 --stderr_to_fd=2 --stdin_to_fd=7 --stdout_from_fd=7 --stdout_to_fd=1 --tty=true --wait_for_connection=false
[==========] 1 test from 1 test case ran. (15190 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] IOSwitchboardTest.ContainerAttach