[DISPATCH-1749] [Test] system_test router function wait_ready does not wait long enough

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Component/s: Tests
• Assignee: Charles E. Rolke
• Reporter: Charles E. Rolke
• Created: 13/Aug/20 21:23
• Updated: 23/Jul/21 18:54

Description

On RHEL/Centos 7 systems sometimes system_tests_policy.VhostPolicyConfigHashPattern sometimes fails with the same pattern as DISPATCH-1689. The VhostPolicyConfigHashPattern test is simply trying to verify that the router processes the config file without exiting due to a config error.

In DISPATCH-1689 the test framework is trying to tear down the router before the router is up. Then the test report shows something like:

RuntimeError: Errors during teardown: 
Process 5978 error: exit code -15, expected 0

Normally the policy test passes. When it fails, though, it prints the tail of the router log file. The log snippet shows that the router is shut down before it prints the server log line

SERVER (info) Running in DEBUG Mode 

Most likely the policy test problem is that the router.wait_ready function opens the listener socket, closes the socket, and returns success so quickly that the router never gets to log the socket open/close and print the rest of the router boot log messages. Once this happens then the test framework falls into the DISPATCH-1689 trap and declares a false failure.

There are several options for making wait_ready wait until the router is really up.

• Grep the log for the "SERVER (info) Running" message
• Require a successful qdstat -g on routers.addresses[0]
• ???

Each of these options has drawbacks.

There are any number of problems in the test suite that could be directly related to wait_ready not waiting long enough.

Issue Links

• relates to: DISPATCH-1689 [test] test framework can't handle qdrouter that is torn down too soon