[MESOS-7036] Rate limiter deadlocks during IO Switchboard-related tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: test
• Labels: flaky mesosphere

Environment

ASF CI

• Assignee: Benjamin Hindman
• Reporter: Greg Mann
• Shepherd: Benjamin Mahler
• Created: 30/Jan/17 23:03
• Updated: 26/Apr/17 17:00
• Resolved: 06/Feb/17 09:36

Description

This has been observed a number of times recently on the ASF CI. While I didn't look through every single failed test log, I've noticed the failure occur during the following tests:

ContentType/AgentAPITest.LaunchNestedContainerSessionWithTTY/1
ContentType/AgentAPITest.LaunchNestedContainerSessionWithTTY/0
IOSwitchboardTest.ContainerAttachAfterSlaveRestart
ContentType/AgentAPITest.LaunchNestedContainerSession/1
ContentType/AgentAPITest.LaunchNestedContainerSessionDisconnected/1
ContentType/AgentAPIStreamingTest.AttachContainerInput/0
IOSwitchboardTest.ContainerAttach
ContentType/AgentAPIStreamingTest.AttachInputToNestedContainerSession/0

In all cases, we see the following:

**** DEADLOCK DETECTED! ****
You are waiting on process __limiter__(518)@172.17.0.3:35849 that it is currently executing.

Find attached an entire example log.

Issue Links

• is related to: MESOS-7065 Deadlock is possible when a pointer to an actor is captured in this actor's callback.