[CASSANDRA-14558] dtest: log-watching thread leak and thread starvation

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Component/s: Test/dtest/python
• Labels: dtest
• Severity: Normal
• Assignee: Stefan Podkowinski
• Reporter: Stefan Podkowinski

Authors

Stefan Podkowinski

Reviewers

Kurt Greaves

• Created: 04/Jul/18 12:29
• Updated: 16/Apr/19 09:29
• Resolved: 09/Jul/18 07:33

Description

We get occasional build timeouts on b.a.o after pytest becomes unresponsive for over 20 minutes. This will result in a thread dump like this one:

https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-2.2-dtest/117/consoleFull

If you look for "Log-watching thread starting" messages and the dump, it becomes quite obvious whats the issue here.

I had a quick look at the python3 / pytest related changes in CASSANDRA-14134 and it seems that some of the handling around dtest_setup's log_watch_thread var has been changed in a way that would prevent eventually yielding the allocated thread.

Issue Links

• links to: WIP-14558